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1  Identification  and  Significance  of  the  Problem  or  Opportunity 

It  is  a  recognized  requirement  in  the  development  path  of  UASs,  by  the  DoD,  FAA  and 
commercial  interests,  that  to  reach  the  next  level  of  autonomy  they  must  be  capable  of  sensing 
their  environment  and  reacting  as  necessary.  As  defined  by  the  JIPT  in  their  recently  published 
draft  UAS  requirements  document,  a  sense  and  avoid  (SAA)  system  encompass  the  following 
functions:  air-to-air,  air-to-ground,  ground-to-ground,  ground-to-air,  and  man  made  hazards. 
Current  and  future  plans  envision  UASs  ranging  in  size  from  the  micro  (approximately  6  inches) 
to  the  macro  (747  FedEx  freighters)  while  their  missions  will  range  from  crop  dusting  to  the 
extreme  in  the  battlefield  to  flight  time  of  a  few  minutes  to  many  days.  In  a  SAA  system,  many 
of  these  functions  can  and  may  overlap  in  technology  implementation  or  may  not  be  required  for 
specific  missions  or  platforms.  Due  to  the  war  and  lack  of  standards,  technology  development 
has  not  kept  up  with  market  need.  A  SAA  system  is  an  enabling  technology.  There  currently 
exist  only  a  few  technologies  that  have  the  capability  and  maturity  to  address  both  the 
operational  needs  and  SWAP  requirements  for  a  SAA  in  the  short  or  near  term.  The  JIPT  has 
estimated  their  timeline  for  full  implementation  and  certification  out  to  2020  as  well  as  RTCA 
SC-203  has  now  estimated  that  their  standards  will  not  be  completed  until  2020.  Flowever,  the 
fact  of  the  matter  is,  that  the  DoD  is  flying  UASs  day-in-and-day-out  without  SAA.  This  provides 
a  clear  path  to  gain  experience  and  deploy  a  SAA  system  that  may  not  include  all  JIPT  functions 
but,  will  enable  new  missions,  safer  missions,  lower  costs  and  manpower  requirements  without 
a  lengthy  certification  process.  The  FAA  has  indicated,  in  open  forums;  1)  that  they  are  open  to 
type  certification  for  a  SAA  system  for  civilian  operation,  and  2)  that  they  expect  the  first  system 
to  achieve  certification  to 
set  the  standard  for  the 
industry". 

To  meet  the  SWAP, 
mission  (both  civilian  and 
military),  and  budgetary 
requirements  of  UAS 
manufacturers,  MilSys 
Technologies  is 
developing  a  core 
collision  avoidance  (CA) 
technology  that  is 
platform  centric  and  can 
be  leveraged  across 
widely  varying  platforms 
and  divergent  missions  (Table  1).  This  will  be  accomplished  by  both  innovative  hardware  and 
algorithm  design.  MilSys  Technologies  believes  that  it  is  possible  in  the  near  term  to  produce  a 
system  that  has  FOV  of  50  to  220°,  weighs  <0.5lbs,  and  has  the  functions  to  perform  air-air,  air- 
ground,  and  autonomous  maneuvering. 

The  goals  of  the  program  were  to  demonstrate  the  feasibility  of  using  both  COTS  hardware  and 
existing  target  detector  and  tracker  technology  for  the  purpose  of  developing  a  SWAP  efficient 
collision  avoidance  system  for  UASs.  The  goals  were  partially  meet  in  that  significant  work  was 
completed  in  both  hardware  and  algorithms,  but  technical  difficulties  arose  and  time  ran  out  on 
the  present  contract,  before  a  full  demonstration  could  be  completed. 

1 .1  Background  -  SAA 

Several  different  technologies  are  now  being  evaluated.  They  range  from  microwave,  radar,  EO 
active,  EO  passive,  and  even  acoustic.  The  type  of  technology  employed  will  primarily  be  a 


Tier  I  -  Dragon 
Eye 

Tier  II  -  Scan 
Eagle 

Tier  Ik/Tier  III 
Global 

Flawk/Predator 

System  1/0 

System  2/1 

System  5/1 

Weight 

<0.5lbs 

<1lbs 

<8lbs 

Size 

0.01  if 

0.03  if 

<0.1  ft^ 

Power 

<5W 

<10W 

<30W 

FOV 

cn 

o 

o 

100° 

220° 

Table  1 :  Potential  configurations  for  various  airframes.  The 
nomenclature  of  X/Y  refers  to  the  number  of  sensors  and  the  need 
for  a  central  track  processor. 
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function  of  the  platform  and  mission.  Each  is  unique  and  no  one  technology  will  cover  all  of  the 
possibilities. 


Currently,  FAA  has  been  directing 
the  regulatory  development  of  SAA 
through  the  1 4  CFR  91:113  Right  of 
way  rules.  Which  to  paraphrase, 
states  that  all  pilots  are  required  to 
“see  and  avoid”  other  aircraft, 
whether  operating  under  Instrument 
Flight  Rules  (IFR)  or  Visual  Flight 
Rules  (VFR),  when  conditions 
permit.  The  FAA  has  empowered 
the  RTCA  and  ASTM  to  develop 
the  regulatory  /standards 
framework  required  so  that  UAVs 
can  be  granted  equal  access,  as 
manned  aircraft,  to  the  national  air 
space  (NAS).  Since  “to  see”  is  a 
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^  R'q)rocessor 

1  Preprocessor  | 

Figure  1 :  Schematic  of  MiiSys  Technologies  Passive 
Coliision  Avoidance  System. 


human  function,  a  natural  limit  on  the  requirements  of  a  SAA  system  is  that  it  must  be  “only  as 
good  as”  a  pilot.  As  one  can  imagine,  as  a  technical  solution  to  the  problem,  the  human  solution 
has  it  limitations.  The  literature  indicates  that  a  diligent  pilot  can  nominally  cover  ±110° 
horizontal  and  ±15°  vertical  not  including  obstructions.  Poor  eyesight,  fatigue,  distractions  and 
the  environment  generally  limit  the  range  of  contact  to  1  to  1 .5  miles  and  12"'  seconds  of 
warning  and  in  most  cases  is  significantly  worse'''. 


1 .2  MilSys  Technologies  Passive  Collision  Avoidance  System  (PCAS) 

MilSys  Technologies  has  been  developing  a  collision  avoidance  system  for  the  past  three  (3) 
years.  The  program  is  currently  funded  under  an  ONR  BAA  (contract  N00014-07-C-0755).  The 
primary  goals  are  to  design,  develop,  and  certify  a  passive  collision  avoidance  system  (PCAS) 
for  UASs.  The  statement  of  work  focuses  on  designing  and  developing  a  man-in-the-loop 
(MITL)  PCAS  based  on  COTS  components  for  UASs  greater  than  17’  wingspan  or  multi-tens  of 
pounds  payload  capacity  (i.e.  Predator).  This  systematic  concept  was  extend  and  downsized 
for  the  application  to  Tier  II  and  Tier  I  UASs,  such  as  Raven,  SilverFox,  ScanEagle  etc.  By  unit 
deliveries,  these  classes  are  the  largest.  Additionally,  part  of  the  funding  was  allocated  to 
advanced  developments  of  future  sensors  and  concepts  which  could  potentially  meet  JIPT 
requirements  within  the  developed  PCAS  architecture.  Overall,  the  effort  is  as  a  multi-year 
hardware  and  software  development  program  culminating  in  flight  certification  in  the  final  year. 

The  technology  is  based  on  uncooled  LWIR  microbolometer  sensors  (MB)  in  a  staring 
configuration  as  proposed  by  the  Priesf  report  of  1998.  The  MBs  will  be  distributed  around  the 
perimeter  of  the  aircraft  with  up  to  360°  of  coverage.  The  total  number  of  sensors  could  range 
between  one  and  ten  (10)  depending  on  the  mission  and  platform.  To  keep  cost  low,  the 
architecture  is  modular  and  distributed  in  order  that  the  largest  number  of  platforms  and 
missions  could  be  accommodated  with  a  minimum  of  customization  (Figure  1). 


From  discussions  with  our  customers,  partners,  and  government  technical  officers,  we  have 
developed  the  following  CA  requirements.  Some  of  the  requirements  are  qualified  by  UAS 
class: 


o 


Must  be  capable  of  providing  CA  and  situation  awareness  (Air-Air)  along  the  aircraft  vertical  ±15' 
and  the  azimuth  according  to  UAS  classification. 

o  Tier  11+  (Predator/Global  Hawk)  -  up  to  360°,  but  more  probably  +1 1 0° 
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o  Tier  1 1  (ScanEagle)  -  ±1 1 0° 
o  Tier  I  (Raven)  -  50° 

•  Must  operate  24/7  and  be  passive. 

•  Sensitivity:  Detect  a  Cessna  type  object  closing  at  250knots  with  >90%  probability  at  limits  of 
detection  (nominal). 

•  Detectability: 

o  Tier  II+  -  4km,  30s,  A30°F,  MLS,  90%  (nominal) 
o  Tier  II -3km 
o  Tier  I  -  2km 

•  For  Tier  II/II+  -  Must  be  capable  of  interfacing  with  TCAS  and  ADS-B  (future). 

•  First  generation  system  must  be  MITL  and  provide  the  ground  operator  with  the  following 
information: 

o  Coordinate  locations  of  objects  of  interest, 

o  Object  prioritization, 

o  Thresholds  declaration. 

•  Future  systems,  as  funding  becomes  available  will  add  progressively  more  functionality  to  meet 
JIPT  requirements, 

o  Airspace  situational  awareness, 
o  Autonomous  maneuver  and  avoid, 
o  Passive  Ranging, 

o  Separation  assurance  around  Class  B,C,D  (airports  etc) 
o  Air-Ground,  Ground-Ground,  Ground-Air, 
o  All  Weather  and  man  made  hazards, 

•  Provide  see  and  avoid  capability  equal  to  or  better  then  current  FAA  requirements  for  14CFR  Part 
91.113  Right  of  Way  Rules 

•  Must  be  retrofitable  and  customizable  so  as  to  fit  into  the  broadest  grouping  of  both  civilian  & 
military  aircraft  covering  the  categories  of  UASs,  rotorcraft,  and  fixed  wing  aircraft. 

•  Must  comply  with  MIL_STD_810F  for  shock,  vibration,  drop,  immersion,  and  temperature  and 
MIL-STD-461/462  for  EMI.  All  relevant  FAA,  military,  aerospace  specs,  as  required. 

•  Meet  all  required  Reliability/Serviceability  Standards  as  defined  by  the  DoD  and  FAA. 

1.3  Technical  Approach 

A  pilot  gauges  a  collision  threat  by  analyzing  its  size,  rate  of  change  (extent  and/or  intensity), 
and  track.  Instinctively,  these  factors  are  prioritized  and  set  to  thresholds,  once  a  threshold  is 
breached,  corrective  action  is  taken. 

There  are  several  operating  assumptions  or  scenarios  that  can  be  useful  in  understanding  the 
operation  of  aircraft  and  the  requirements  of  a  CA  system: 

In  the  NAS  there  are  two  (2)  types  of  traffic:  cooperative  and  non-cooperative  while  in 
the  battlefield  all  traffic  is  assumed  to  be  non-cooperative.  Cooperative  traffic 
broadcasts  its  position  using  a  transponder  (TCAS  or  ADS-B  future).  Non-cooperative 
traffic  does  not. 

Additionally,  according  to  14  CFR  Part  91 ,  aircraft  that  operate  below  250  knots  must 
remain  below  10,000  feet  and  any  aircraft  flying  above  10,000  feet  must  have  a 
transponder.  Therefore  in  the  NAS,  fast  aircraft  only  fly  >250  knots  above  1 0,000  feet, 
allowing  a  TCAS  enabled  system  to  provide  the  additional  detection  range  (Table  2).  In 
the  battlefield,  the  condition  of  lower  and  slower  still  applies  but,  is  not  universally  true. 

Above  1 8,000  feet  aircraft  (Class  A)  are  required  to  fly  under  IFR  (instrument  flight  rules) 
and  be  under  ATC  (air  traffic  control).  Therefore,  theoretically,  when  operating  under 
IFR,  compliant  aircraft  will  not  collide.  However,  several  mid  air  collisions  have  taken 
place  when  aircraft  are  not  compliant  such  as  the  Tu-154  /DHL  mid  air  collision  and  the 
AIR  RAGE  Airbus  mid  air  collision. 
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The  two  (2)  examples  highlighted  above  demonstrate  the  need  a  need  for  a  platform  centric  CA 
technology  over  and  above  the  existing  implementations  of  TCAS,  ADS-B  and  ATC  in  Class  A 
airspace. 

MilSys’  approach  is  to  mimic  a  pilot’s  analytical  approach  with  a  technological  solution.  Simply 
speaking,  the  system  operates  as  a  hot  spot  detector  and  corrective  action  must  be  taken  when 
an  object  breaches  a  threshold  and  remains  stationary  in  the  field  of  view.  Proprietary  models, 
algorithms  and  methods  are  being  developed  to  sense,  detect,  cluster,  discriminate  and 
threshold  objects  of  interest  (See  Sections  5).  Thresholds  will  be  determined  by  the  altitude, 
speed,  external  temperature,  atmosphere/weather  etc.  based  on  the  platform  and  its  mission 
profiles.  This  approach  can  meet  the  SAA  requirements  for  both  civilian  and  military  aircraft. 

The  process  to  avoid  collision  can 
be  broken  into  three  (3)  distinct 
phases;  detection  and  tracking, 
path  planning,  and  maneuvering. 

Detection  and  tracking  is  the  period 
of  time  between  the  initial 
breaching  of  a  threshold  and  the 
determination  of  its  track.  Path 
planning,  in  a  MITL  system,  is  a 
function  of  the  speed  of  the 
up/downlink  and  response  of  the 
ground  controllers.  In  a  UAS  with 
autonomous  avoidance  guidance 
path  planning  would  be  the  time  to 
determine  a  de-confliction  plan. 

The  time  to  maneuver  is  a  function 
primarily  of  the  UAS  bank  angle. 

Nominally,  the  UAS  of  the  Predator  Class  will  take  1  seconds  to  maneuver  500ft  at  a  bank 
angle  of  1 8°  independent  of  speed.  In  smaller  Tier  I  UAS  the  time  could  be  as  short  as  5 
seconds.  Independent  of  the  platform,  the  detection  and  tracking  of  objects  will  be  in  the  range 
of  2-5  seconds.  All  of  these  times  are  generally  fixed,  with  the  caveat  that  the  UAS  could 
maneuver  faster  if  it  increased  its  bank  angle.  The  gating  factor  is  the  MITL  which  has  been 
specified  by  UAS  manufacturers  as  15  seconds.  From  a  systems  perspective,  the  goal  is  to 
minimize  MITL  delays  while  providing  the  maximum  amount  of  detection  and  tracking  time.  The 
object’s  slant  range  to  the  UAS  is  a  key  factor  in  maximizing  the  MITL  time.  Front  on 
approaches  will  have  a  much  shorter  MITL  time  than  side  on.  Overall,  this  sets  the  detection 
time  to  be  32  seconds  out  for  the  larger  Tier  ll-i-  UASs  and  as  short  as  1 0  seconds  for  Tier  I. 

Currently  under  the  statement  of  work  for  the  BAA,  PCAS  is  a  MITL  system  for  Predator  Class 
UASs.  As  funding  becomes  available,  and  as  the  regulations,  standards  and  technology 
progress’,  its  MilSys’  intention  to  integrate  additional  amounts  of  autonomy.  This  path  may  be 
as  follows:  MITL,  autonomous  path  planning  and  recommend  maneuvering,  independent 
emergency  action,  limited  autonomy,  and  full  autonomy. 

Discussions  with  the  smaller  Tier  I  and  Tier  II  manufacturers  and  government  program 
managers  indicated  a  need  and  desire  to  move  more  quickly  to  autonomous  path  planning  and 
maneuvering.  The  operational  plan  would  be  to  be  under  MITL  control  for  Class  B,C,D  airspace 
on  landing,  takeoff  and  when  transiting.  However,  when  in  Class  E  airspace  during  transit  and 
on  station  the  UAS  would  operate  under  full  autonomy.  The  rational  being  that  >99.99%  of  the 
time  this  airspace  is  completely  void  of  any  traffic. 


Altitude 

Table  2:  Factors  effecting  collision  times 

>17,000 

In  the  NAS,  operating  in  Class  A, 

ATC  controlled  airspace 

Low  < 10,000ft 

Lower  Airspeeds  <250knots 

Lower  Temperatures  Differentials  (<A°F) 
Smaller  objects 

More  atmospheric  absorption  (>a) 

No  TCAS 

High  >1 0,000ft 

Higher  Airspeeds  >250knots 

Higher  Temperature  Differentials  (>A°F) 
Less  atmospheric  absorption  (<a) 

Larger  objects 

TCAS  (NAS  only) 
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1 .4  Operational 

The  PCAS  architecture  has  been  designed  from  the  ground  up  to  be  SWAP  and  cost  efficient, 
as  well  as  modular  and  highly  flexible  in  order  to  meet  the  requirements,  both  new  and  a  retrofit, 
of  most  aircraft  (UAS,  manned  fixed  wing  and  rotorcraft)  on  the  market  today.  It  consists  of  two 
(2)  hardware  components:  sensors  and  a  track  processor  (TP).  To  reduce  development  time, 
costs,  and  program  risk,  both  the  sensors  and  TP  will  be  based  on  COTS  or  modified  COTS 
components. 

The  sensors  will  operate  as  independent  self  contained  units  having 
hardware/software/firmware  for  image  stabilization  (IMU),  control,  communication,  an  IR  (or 
visible  as  required)  camera  with  lens,  and  proprietary  sense,  detect  and  threshold  algorithms. 
The  algorithms  will  be  based  on  LMS  type  filtering  (5.1)  not  optical  flow.  Optical  flow  being 
determined  as  to  processor  intensive  for  this  type  of  system.  The  FOV  would  be  tailored 
depending  on  sensor  location,  UAS  class  and  mission  profile,  with  the  front  sensor  having  the 
smallest  FOV  and  in  the  rear  the  largest.  One  proposed  configuration  is  «20°  at  the  nose,  50° 
at  the  wing  tips  and  >100°  at  the  rear.  They  will  be  “plug  and  play”  with  the  TP  allowing  easy 
customization. 


Figure  2:  Flow  diagram  of  the  image  through  the  sensor. 

The  image  captured  by  the  lens  and  FPA  (Figure  2)  will  be  processed  by  the  internal 
DSP/FPGA  and  firmware  to  correct  for  internal  calibration  errors,  normalization,  dead  pixel 
removal  etc.  The  DSP/FPGA  will  then  perform  an  external  calibration  (as  required)  and  stabilize 
the  image  as  required  from  information  provided  by  the  IMU.  This  processed  image  will  then  be 
passed  through  the  detector  and  threshold  algorithms.  As  required,  detection  reports  (Figure  3) 
will  be  produced  and  communicated  with  TP.  The  image  will  then  be  discarded. 


The  central  brain  (i.e.  TP)  will  be  the  locus  of  the  system  (Figure  1).  It  will  control  and 
communicate  with  the  sensors, 
the  autopilot,  the  operator  (pilot 
or  ground  control)  and  the 
aircraft.  It  will  receive  detection 
reports,  inputs  from  TCAS,  INS, 
and  other  relevant  data  from  the 
aircraft  or  ground  control. 

Proprietary  algorithms  (See  5) 
will  translate  detection  reports 
into  tracks  to  be  monitored,  prioritized  and  set  to  thresholds  (Figure  4).  MITL  or  autonomous 
algorithms  will  make  the  appropriate  decisions  and  control  the  aircraft  to  avoid  any  threat 
conditions. 


Figure  3:  Example  of  a  detection  report. 


A  proposed  configuration  for  the  Predator,  with  five  (5)  sensors  and  TP,  will  have  a  SWAP  of 
nominally  30Watts,  <8lbs,  and  <0.1  cubic  feet. 
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Figure  4:  Block  diagram  of  some  of  the  processing  algorithms. 

2  System  Modeling 

A  system  model  which  will  allow  the  prediction  of  range  performance  based  on  system  design 
inputs,  target  geometries,  thermal  profiles  and  operating  environmental  conditions  was 
developed  by  Magnolia  Optical  under  the  leadership  of  Dr.  Ashok  Sood.  The  model  is  designed 
to  answer  this  type  of  question:  Given  a  microbolometer  with  an  f/1  optics,  25mm  diameter, 
17um  pixel,  etc.  operating  at  mid  latitude  summer  (MLS)  at  what  range  will  it  have  a  50% 
probability  of  detecting  a  Cessna  type  target  (4m^,  A30°F)  The  validity  of  the  data  will  be 
compared  against  NVTherm2002  and  NVThermlP,  as  well  as  data  collected  in  the  field. 

Many  of  the  microbolometer  manufactures  have  contributed  their  actual  production  line  data  for 
comparison. 

2.1  Bolometer  sensor  model(s)  overview 

2.1 .1  Excel  model  for  single  detector  pixel  performance  prediction 

This  excel  model  was  created  to  provide  accurate  estimations  of  bolometer  array  performance 
parameters,  notably  NETD  and  MRTD,  for  the  many  types  of  devices  currently  in  production. 
Inputs  include  basic  materials  properties  for  the  bolometric  material,  the  absorber  layer  if  used, 
the  metallizations,  and  the  structure  of  the  pixel  (table  and  leg  structure).  Electrical  resistances, 
TCR’s,  heat  capacitances  and  thermal  conductances  are  calculated  and  are  used  to  predict  the 
time  constant,  the  response  and  the  NETD  of  the  device.  Inputs  also  include  the  optical  system 
and  the  biasing  and  readout  characteristics.  Performance  results  can  be  seen  in  Figure  5,  and 
optics,  scene  and  electronics  inputs  are  shown  in  Figure  6.  The  excel  sheet  is  set  up  to  easily 
allow  inputs  for  three  VOx  manufacturers’  and  the  alpha-silicon  manufacturers’  devices  (Figure 
9).  NETD’s  here  are  calculated  for  the  four  major  noise  sources  (Johnson  noise,  thermal 
fluctuation  noise,  1/f  noise  and  read  noise). 
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Figure  5:  Excel  performance  chart 
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Figure  6:  Scene,  optics  and  eiectronics  inputs 

Shown  are  the  noise  voltages,  the  NEP,  the  NETD  calculated  from  the  heat  balance  equation, 
the  current  noise,  the  noise  equivalent  flux  density,  the  noise  voltage  calculated  from  a  variety  of 
expressions  found  in  literature  sources'''  ^'.  Figure  5  shows  that  near  exact  NETD  agreement 
was  accomplished  from  three  independent  equations  and  methods  (NETDsw,  NETDrg,  and 
NETDhbef  The  slightly  higher  NETD’s  from  the  heat  balance  equation  (NETDhbe)  are 
probably  closer  to  reality  than  those  predicted  purely  from  equations  as  the  response  used  in 
most  equations  is  an  approximation  and  assumes  that  the  thermal  leg  conductance  is  much 
greater  than  the  radiative  conductance. 

2.1 .2  Imaging  matlab  model  for  aggregate  pixel  (FPA  and  sensor)  performance 
prediction 

A  model  in  matlab  for  the  generation  of  dynamic  image  sequences  was  developed  to  help 
assess  and  characterize  sensors  and  cameras  by  providing  images  of  bar  targets  in  user- 
controlled  noise  backgrounds.  This  model  is  used  to  simulate  dynamic  bar  target  (NETD  and 
MRTD)  tests.  The  model  generated  bar  target  images  of  user-defined  delta  temperature  and 
size  are  shown  at  30Flz  as  in  an  actual  test  with  a  physical  bar  target,  collimator,  display  and 
human  eye.  Temporal  and  spatial  noise  can  be  added  by  the  user,  the  bars  can  be  blurred  by 
the  diffraction  PSF  or  a  user-defined  Gaussian  PSF,  and  are  displayed  in  real-time.  The  images 
are  first  constructed  at  super-resolution  (8x  to  32x),  convolved  with  a  super-resolution  PSF  then 
binned  down  to  the  FPA  format  of  interest.  Initial  results  (observer  MRTD’s)  are  much  lower  at 
the  lower  spatial  frequencies  than  the  NETD  values  and  these  MRTD  plots  agree  with  the 
NVThermlP  run  outputs.  Older  MRTD  equations  tend  to  overestimate  the  MRTD  as  does 
NVTherm2002.  MRTD(f)  will  be  an  important  parameter  in  calculating  probabilities  of  detection, 
identification  and  recognition.  For  bar  target  spatial  frequencies  that  approach  the  detector 
spatial  frequency  (Nyquist),  large  variation  in  MRTD  is  a  result  of  aliasing.  Placement  of  the 
bars  in  exact  alignment  gives  the  lowest  MRTD;  however,  in  reality  this  is  almost  never  the 
case.  As  the  spatial  frequency  approaches  1/(2*0det),  bar  location  becomes  more  important 
and  large  variations  in  MRTD  are  seen  for  just  small  changes  (portion  of  a  detector  pixel)  in  bar 
position. 
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Figure  7:  Simulated  bar  target  images  from  the  matlab  program 

These  simulated  MRTD’s  and  MRTD’s  calculated  by  NVTherm  or  from  NVTherm  expressions 
will  be  used  to  predict  the  N50  spatial  frequency  and  range  and  the  corresponding  identification 
and  recognition  probabilities  and  ranges  in  section  2.3.3. 

2.1 .3  Capacity  for  sequential  vendor  model  inputs  and  fair  comparison 

The  model  has  been  adapted  to  get  fast  and  accurate  results  for  the  four  manufacturers’ 
bolometers.  The  user  can  switch  between  the  four  by  simply  choosing  a  number  from  1  to  5  in 
the  “choices”  cell  shown  in  Figure  8.  Once  the  sheet  is  populated  by  vendor’s  specifications  or 
our  best  approximations,  comparisons  of  bolometer  arrays  in  identical  optics  and  for  identical 
operating  conditions  can  be  accomplished.  Here  the  model  may  also  be  used  for  performance 
optimization  since  we  have  knowledge  of  all  the  noise  sources  and  the  response.  Bias  voltage, 
sample  time,  operating  temperature  can  all  be  varied  as  well  as  the  characteristics  of  the 
bolometer. 
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Figure  8:  bolometer  choice  section 


2.1 .4  Iterative  heat  balance  equation  and  comparison  to  approximations  often  used 

For  a  more  realistic  representation  and  to  assure  that  thermal  runaway  is  not  occurring  at  the 
biases  applied,  the  heat  balance  equation  is  solved  for  the  baseline  conditions  (no  target),  and 
the  new  conditions  (target  on  a  pixel).  The  model  iterative  solves  for  these  conditions,  finds  the 
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bolometer  temperatures,  calculates  resistances  and  currents,  signal  to  noise,  and  NETD.  The 
model  supplies  a  user-defined  target  temperature.  Here  it  was  set  at  1 K  above  ambient.  The 
model  calculates  the  signal  and  the  signal  to  noise.  This  is  then  divided  into  the  delta  T  value  to 
get  the  NETD.  These  values  have  corresponded  closely  to  the  purely  equation  derived  NETD’s 
but  show  consistently  higher  NETD’s  due  to  the  lower  response  calculated  from  the  heat 
balance  equation.  This  lower  response  (generally  10-20%  lower)  is  expected  to  be  closer  to 
reality  than  the  response  approximation  usually  used  which  does  not  consider  the  effects  from 
radiative  loss,  only  from  the  leg  conductance.  For  higher  leg  conductance  to  radiative 
conductance  ratios,  the  responses  become  equal.  The  SNR  and  computed  NETD  from  the  heat 
balance  equation  were  found  to  be  in  close  agreement  with  the  theoretical  models  (equations) 
for  a  large  variety  of  bolometer  characteristics. 
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Figure  9:  Heat  balance  equations 

2.2  Sensor  systems  model  and  evaluation  criteria 

2.2.1  Mathematical  basis  for  modei:  NETD,  NEP,  MTF,  CTF,  and  MRTD  equations 
compiied  and  derived  from  iiterature  and  from  NVTherm  modeis 

NETD  and  NEP  derivations:  The  bolometer  NETD  consists  of  contributions  from  background 
temperature  fluctuation,  bolometer  temperature  fluctuation,  electrical  current  Johnson  noise, 
bolometer  1/f  noise  and  readout  noise.  The  NETD  in  relation  to  the  NEP,  response  and  D-star 
is: 


4/72  y  4^2 

NETD  =  NEP - =  - 

TMdM/dT]^,  zMdM/dT]^ 


4E^  4^ 

rX\dMldT\^,  D* 


where  F  is  the  optics  f/number,  and  dM/dT  is  the  spectral  radiance  contrast  of  the  scene 
evaluated  over  the  spectral  band 

The  NETD  components  are: 


NETD  Johnson: 

NETD  1/f: 

NETD  thermal  fluct: 


NETD.  =[ 


^  ^  a  V 

“  hin 


4/72 


dM/dT 


NETD,,.  =[— ]  [k,  Hf2  //i)f '  [ 


4/72 


a,. 


^d^ib^o  dM/dT 


NETD.  =[4kT^Gf^.f^  [ 


4/72 


eff- 


dM  / dT 


feff  =^VfbolJread^ 
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NETD  background  fluct: 


netd,  [ 


cc,. 


K^ib'^o  dM  I  dT 


(negligible) 


Note  that  NETD  varies  strongly  with  two  commonly  varied  parameters,  the  f-number  and  the 
background  temperature  (and  temperature  contrast  dM/dT).  NETD  improves  with  slower  optics 
as  shown  in  Figure  10a.  NETD  degrades  at  cooler  background  temperatures  (tracking  with  the 
spectral  radiant  contrast)  as  shown  in  Figure  10b.  This  NETD  degradation  (increase)  observing 
cool  scenes  can,  in  part,  be  compensated  for  as  bolometer  noise  may  improve  and  response 
(TCR)  may  increase.  L-3  has  claimed  improved  performance  for  a-Si  operating  at  below 
ambient  temperatures.  They  report  about  a  2x  improvement  in  NETD  from  300K  to  220K 
operation  due  mostly  to  an  increase  in  TCR  and  resistance. 
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Figure  1 0:  NETD  as  a  function  of  f-number  (left)  and  of  scene  temperature  (right) 

NETD  and  MRTD  are  the  commonly  used  metrics  for  extended  resolved  target  evaluation; 
however,  for  subpixel  targets  the  NEP  and  NEI  are  used. 

MTF’s  and  CTF’s:  MTF’s  used  in  the  model  include  the  diffraction  MTF,  the  detector  spatial 
sample  MTF,  the  eye  MTF  (used  in  some  expressions),  and  the  squeeze  or  spurious  response 
MTF.  These  have  been  compared  to  the  MTF  expressions  used  in  both  NVTherm  models  and 
are  identical  as  are  the  output  plots.  Shown  in  Figure  1  la  are  these  and  some  of  the  additional 
MTF  types. 
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Figure  1 1 :  MTF’s  vs.  spatial  frequency 

CTF  system  and  CTF  eye:  The  contrast  threshold  function  is  related  to  the  MRTD 
(CTF=MRTD/(2*Stmp)),  where  Stmp  is  the  scene  temperature  contrast.  The  Stmp  default  value 
in  NVTherm  is  4C  which  is  the  value  used  in  my  excel  and  matlab  model  runs.  The  CTF  plots  in 
Figure  11b  are  from  the  eye  CTF  equations  given  in  the  NVTherm2002  and  NVTherm  IP 
manuals. 


2.2.2  System  level  analysis  (bar  target  imaging  and  MRTD  prediction) 

About  a  dozen  different  MRTD  expressions  were  found  in  the  literature  dating  back  to  the  early 
1980’s.  Past  expressions  were  not  found  to  be  accurate  over  the  entire  spatial  frequency  range. 
Some  of  these  have  been  incorporated  into  the  model  and  the  user  may  choose  which  to  use. 
Plots  from  these  expressions  are  shown  in  Figure  12  and  Figure  14  along  with  NVTherm  IP  run 
results  and  a  visual  assessment  of  matlab  simulated  images  shown  to  the  observer  at  30  FIz. 
Expressions  are  from  the  Patches  and  Lloyd  papers,  and  the  two  latest  versions  of 
NVTherm'‘''''‘'\ 


The  Lloyd  expression  which  gives  a  pretty  good  fit  in  the  mid-range  spatial  frequencies  is: 


MRTD  = 


SNRt 

- (— ) 

1.51  > 


NETD 


H, 


^det/. 


where  SNRt=2.25 


Adding  the  second  constant  term  to  the  expression  improves  the  low  frequency  accuracy.  This 
constant  corresponds  to  the  eye  and  display  contrast  limitations. 


MRTD  =  [{ 


SNRt  A ^2 

- (-) 

1.51  \ 


NETD 

H, 


+[MRTD^,}^r" 


The  MRTD  NVTherm2002  expression  given  in  the  manual  is: 


MRTD(fJ  = 


CTF,, 

M,Hf  ^\dL/dT)D;,Tj(2tJ, S, 


fsf  + 


} 


1/2 


where  Md  is  the  magnification  of  display,  Hf  is  the  combined  MTF,  dL/dT  is  the  scene  spectral 
radiant  contrast,  G  is  the  display  gain,  Keye=823,  (BwBl)^'^/Sl  is  the  eye  filter  response 
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Substituting  for  the  D-star  to  get  the  expression  in  terms  of  NETD  we  get: 


xl/2 


MRTIXf, )  =  { [—  K 


Sl 


t 

Cr 


CTF  (n  R  4S^ 

The  MRTD  from  NVTherm  IP  is:  MRTD(f)  = - ^{r2ar.  ^  ^ 

4F^ 

where  rdet  = - - - 

7tTj{dLldT)Dl,{t,tJtff'^ 


and  in  terms  of  NETD  is: 


CTF 

MRTI\f^)  =  -^ 
M,H 


{[2V2, 


a- 


(Vl) 


1/2 


t  AS 

-{^y^NETDO^J  +  — 

G 


tmp  jl/2J 


This  last  expression  and  the  Lloyd  expression  give  the  best  MRTD  fits  to  data  (simulated  bar 
target  images)  and  with  the  results  from  NVTherm  IP  runs.  These  results  are  described  more 
fully  in  the  model  validation  section.  These  equations  were  used  in  the  excel  model  and  images 
were  generated  in  an  identical  matlab  version  of  the  model  to  generate  realistic  bar  targets  with 
temporal  noise.  These  images  were  compared  to  MRTD  predictions  from  this  excel  model  and 
to  NVTherm  outputs.  Some  representative  images  and  MRTD’s  curves  are  shown  in  Figure  12 
and  Figure  14. 
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Figure  12:  MRTD’s  from  theory  (top  blue  and  yellow),  from  NVTherm  IP  runs  (black  diamonds), 
and  discerned  from  matlab  image  sequences  shown  at  30Hz  (red  circles);  three  images  at 
1 .25*NETD  bar  target  temperature  and  summed  over  the  eye  integration  time  are  shown  at  bottom 

right. 
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2.2.3  MTF,  CTF,  target  temperature  contrast  and  probabilities  of  detection,  recognition 
and  identification 

In  addition  to  the  NETD,  the  MTF  and  CTF  are  critical  parameters  in  predicting  MRTD.  The 
main  components  of  the  MTF  are  from  optics  diffraction,  detector  sampling  and  aliasing.  The 
CTF  is  determined  from  the  human  eye  and  the  display  characteristics  such  as  the  display 
brightness  and  magnification  and  the  human  eye’s  response  to  light  level  and  spatial  frequency. 
MTF’s  and  CTF’s  are  shown  in  Figure  13  for  a  typical  LWIR  f/1  bolometer  based  sensor.  The 
sheet  consists  of  the  target  section,  the  probabilities  section,  the  MRTD  sections,  and  the  MTF 
section.  Other  parameters  of  interest  include  the  D-star,  the  NEP,  the  CTF’s  and  the  SNR 
where  the  SNR  is  calculated  for  the  target  specified  in  the  target  section  and  the  atmosphere 
(radiance  and  transmittance)  specified  for  the  range  and  altitude. 
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Figure  13:  Main  sheet  for  sensor  performance  evaluation 

2.3  Model  validation 

2.3.1  NETD  comparisons  to  literature  and  other  model  results 

Expressions  from  a  variety  of  models  for  the  noise  and  NETD  were  found  in  the  literature.  The 
expressions  from  the  various  sources  were  slightly  different.  Each  set  of  equations  had  some 
shortcomings  or  approximations  that  were  explored.  More  accurate  expressions  for  the  optical 
throughput  and  for  the  response  were  developed.  For  example,  a  common  expression  for  the 
optical  throughput  involves  the  expression  4F^  or  4F^+1 .  An  exact  expression  was  found  whose 
solutions  lie  between  the  values  from  these  two  expressions. 

The  scene  solid  angle  is  derived  from:  q  =  2.t 

Vl  +  1/(4F") 
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The  model  is  in  agreement  with  three  independent  models  (expressions  and  outputs)  for  noise 
and  NETD.  For  the  NETD  calculations,  two  responses  were  used,  the  heat  balance  equation 
response  and  the  thermal  leg  conductance  response. 

The  heat  balance  equation  response  is:  ^Hte=(yto-yto)liP:c-Psc) 

oc  V  R  R 

The  leg  conductance  approximation  is:  9?^  =  ^ ^  ^  ^ 

G  iRi,+Ri^y 

The  approximation  overestimates  the  response  by  about  10%  and  underestimates  the  NETD 
also  by  1 0%  for  these  f/1  microbolometer  cases. 

2.3.2  MTF  and  MRTD  comparisons  to  NVTherm2002  and  NVTherm  IP 

In  general,  the  MRTD  is  equal  to  the  NETD*k2  /  MTF.  The  constant  k2  actually  has  a  frequency 
dependence  that  slightly  increases  the  MRTD  at  the  lower  spatial  frequencies.  In  NVThermlP, 
the  contrast  transfer  function  (CTF)  includes  this  dependency  as  can  be  seen  in  the  NVThermlP 
data  points  in  Figure  14.  The  MRTD  plot  from  the  new  NVThermlP  CTF  expression  agrees  well 
with  the  matlab  image  evaluations.  The  MRTD  plot  and  the  target  effective  temperature  are 
used  in  NVTherm2002  to  find  the  N50  point  (target  spatial  size  where  the  probabilities  of 
detection  /recognition  /identification  are  50%).  NVTherm  IP  uses  the  system  CTF  and  the  target 
temperature  contrast  to  find  the  N50  point  (renamed  in  NVTherm  IP  as  the  V50  point).  These 
are  just  the  previous  NVTherm2002  parameters  divided  by  2*Stmp,  where  Stmp  is  the  scene 
temperature  contrast  which  is  assumed  in  these  simulations  to  be  4C.  The  MTF’s  currently 
used  in  the  model  include  diffraction,  detector  shape,  display,  and  eye  MTF’s  as  shown  in 
Figure  12a.  For  the  MRTD  calculation,  the  NVTherm2002  model  used  the  system  MTF  which 
included  the  eye  MTF  and  a  constant  (SNRt).  The  newer  version  uses  the  system  CTF  and  the 
system  MTF,  but  without  the  eye  MTF.  For  this  47mK  NETD  bolometer,  the  MRTD  is  seen  to 
reach  a  minimum  of  about  16mK  in  the  0.40-0.60  c/mrad  spatial  frequency  range  as  predicted 
by  NVTherm  IP  (red  diamonds).  This  matches  the  14mK  MRTD  discerned  from  the  30Hz 
matlab  simulated  images  (green  squares).  The  difference  between  the  plots  modeled  in  the 
excel  program  and  the  data  plots  (NVTherm  run  outputs  and  visual  MRTD  data)  occurs  because 
of  differences  in  the  eye  CTF  curves  used  and  the  positive  effects  of  aliasing  which  improve  the 
actual  MRTD  at  the  higher  frequencies.  The  excel  model  MRTD’s  do  not  currently  provide  for 
spurious  response  and  for  the  case  where  the  bars  are  perfectly  aligned  with  the  detector  pixels 
(0  degree  phase  case).  When  considering  these  factors,  the  MRTD  is  actually  fairly  flat  until  the 
spatial  frequency  approaches  the  Nyquist  frequency.  The  model  which  uses  the  eye  CTF 
expression  alone  overestimates  the  MRTD  at  high  spatial  frequencies. 
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Figure  14:  MRTD’s  from  equations  (biue  and  purpie),  NVTherm  iP  outputs  (red  diamonds),  matiab 
image  viewing  (green  squares),  ieft  piot  is  without  Stmp  component,  right  is  with  fuli  expression 

2.3.3  Detection,  identification  and  recognition  of  smaii  (subtended)  objects 

Recently  we  procured  copies  of  NVTherm2002  and  NVThermlP  for  comparisons  with  this 
model’s  system  level  outputs.  The  cooled  and  uncooled  detector  models  in  the  NV  products  are 
not  at  all  comprehensive;  however,  their  image  evaluation  models  are  considered  one  of  the 
standards  for  predictions  of  MRTD’s  and  detection  and  ID  probabilities.  Many  of  the  MTF  and 
MRTD  equations  I  used  in  my  model  were  taken  from  or  derived  from  equations  in  the  NVTherm 
manuals.  As  such,  exact  agreement  with  NV  model  products  for  system  level  parameters  is 
expected.  Much  of  the  work  in  this  model  is  geared  towards  providing  a  physics-based 
simulation  tool  for  the  accurate  approximation  of  NETD  for  the  many  material,  optical  and 
electronic  properties  that  are  involved  in  each  of  the  manufacturers’  products.  The  model  can 
then  be  used  as  a  tool  for  device  geometry  and  device  materials  optimization  and  with  the 
addition  of  the  image  generation  and  analysis  functions  will  provide  an  end-to-end  uncooled 
sensor  analysis  tool. 

The  next  task  was  to  compile  as  extensive  a  database  of  bolometer  pixel  characteristics  as 
possible  from  the  vendors  inputs,  input  these  into  the  model,  run  performance  predictions  and 
compare  these  to  data  if  available,  and  extend  the  image  evaluation  section  (MTF’s,  MRTD’s, 
CTF’s)  and  assure  agreement  with  NVTherm  outputs.  Accomplishing  these  tasks,  the  model 
was  able  to  provide  validated  performance  predictions  from  basic  semiconductor  materials 
inputs  all  the  way  through  to  image  and  image  sequence  generation  and  image  evaluation. 

Bolometer  sensor  system  performance  parameters  such  as  probabilities  of  target  detection, 
target  identification  and  target  recognition  are  based  on  their  mathematical  components  such  as 
the  NETD,  the  MRTD,  and  the  target  thermal  contrast  and  size  (extent  on  the  FPA  in 
cycles/radian). 

The  NETD  is  calculated  from  bolometer  material  and  geometrical  characteristics  (pixel  and  leg 
geometries,  electrical  and  thermal  characteristics)  from  vendor  specifications.  The  main 
parameters  are  the  pixel  heat  capacity,  the  leg  thermal  conductance,  the  sensing  material’s 
absorptance,  resistance,  and  TCR.  Noise  is  also  a  factor  and  consists  of  a  combination  of 
Johnson  noise,  thermal  fluctuation  noise,  background  fluctuation  noise,  and  1/f  noise.  We  also 
employ  controlled  spatial  noise  (3D  noise)  in  the  model  to  account  for  row  and  column  noise. 
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etc.  This  model  was  based  on  the  Night  Vision  3D  noise  model.  The  final  NETD  will  consist  of 
the  traditional  temporal  NETD  value  plus  the  spatial  noise  components. 

The  probabilities  and  ranges  are  calculated  from  the  MRTD  and  the  target  thermal  contrast  and 
size  (cycles  on  target  from  Johnson  criteria).  Also,  for  longer  ranges,  we  need  to  add  the 
atmospheric  transmittance  into  the  equation.  The  model  finds  the  intersection  of  the  MRTD 
curve  and  the  target  thermal  contrast  (apparent  target  contrast)  curve  on  the  plot  of  temperature 
(y-axis)  vs.  spatial  frequency  (x-axis).  MRTD  and  target  contrast  are  also  plotted  against  range 
where  we  diminish  the  target  contrast  by  the  atmospheric  transmittance  as  a  function  of  range. 
The  intersection  of  the  two  curves  gives  us  a  spatial  frequency  value  that  is  used  in  the 
detection,  identification  and  recognition  equations  to  determine  their  probabilities.  This 
calculation  is  also  done  in  the  NVTherm  model  and  has  been  compared  to  this  model’s  findings. 
Thus,  knowing  the  target  size  and  thermal  contrast,  we  find  solutions  of  the  probability 
equations  and  the  range  equations  for  that  ideal  target  on  the  modeled  bolometric  sensor. 

For  subpixel  sized  targets,  the  metric  to  use  is  the  Noise  Equivalent  Irradiance  (NEI)  or  the 
Noise  Equivalent  Power  (NEP).  For  pixel  sized  targets  the  NETD  value  can  be  used  to 
determine  detection.  For  extended  targets,  the  MRTD  and  target  size  and  contrast  are  required. 
The  NEI  and  NEP  expressions  use  the  target  subtended  area  and  irradiance,  the  detector 
response  and  noise,  and  the  detector  optical  efficiency  which  is  mainly  determined  by  the  fill 
factor  and  the  diffraction  spot  size.  The  NETD  is  calculated  from  the  bolometer  and  optics 
properties  discussed  previously.  The  probabilities  based  on  the  MRTD  and  the  target 
characteristics  are  more  involved  and  are  open  to  some  interpretation.  NVTherm2002  has  been 
updated  and  the  new  NVThermlP  uses  slightly  different  expressions,  such  as  the  CTF  (contrast 
threshold  function)  and  introduces  the  V50  expressions  to  replace  the  formerly  used  N50 
expressions. 

As  an  example  that  illustrates  the  detection,  identification  and  recognition  parameters,  we 
consider  a  typical  uncooled  sensor.  The  format  is  640x480  with  17  micron  pixels,  the  optics  is 
f/1  (1 0  cm  aperture),  and  the  frame  rate  is  30Flz.  The  fill  factor  is  75%  and  the  measured  NETD 
is  30mK.  The  FOV  is  4.7  x  6.2  degrees  and  the  IFOV  is  0.17  mradian.  The  MRTD  curve  here 
has  been  generated  using  a  derivation  of  the  NTherm2002  manual  expression 


CTF 

MRTD= - ^{[ 


4  S, 


t  4S'^ 

-K(^y'^NETDe,J,f 


The  apparent  target  temperature  is:  7;^^  =  (r)  and  was  assumed  to  be 

0.1  K. 


In  this  example  we  do  not  include  the  atmospheric  transmittance  which  is  close  to  unity  for  the 
shorter  ranges  under  clear  conditions.  From  figure  12,  the  intersection  of  the  target  with  an 
apparent  target  temperature  of  lOOmK  and  the  MRTD  curve  occurs  at  a  spatial  frequency  of  3.5 
to  4.0  cycles/mradian  that  correspond  to  a  range  of  5.2  to  6.0  kilometers.  This  spatial  frequency 
is  defined  as  the  cutoff  frequency  and  is  the  spatial  frequency  that  specifies  the  50%  probability 
of  recognition  for  this  sensor  and  target.  The  MRTD  curves  shown  are  from  derivations  of 
Lloyd’s  expression  and  from  NVTherm  expressions.  The  sporadic  points  are  from  visual 
inspection  of  images  generated  in  the  matlab  program. 
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Figure  15:  MRTD  curves  (temperature  vs.  spatial  frequency  and  vs.  range)  for  a  4x4  meter  target 

The  recognition  probability  for  other  spatial  frequencies  can  be  calculated  from  the 
NVTherm2002  expression:’'’" 

Much  of  the  probability  section  of  this  model  is  based  on  NVTherm2002  derivations  which 
employ  N50  values  and  traditional  Johnson  criteria  using  MRTD’s  and  apparent  target 
temperature.  The  newer  version,  NVThermlP  uses  V50  criteria,  the  apparent  target  temperature 
contrast,  and  the  contrast  theshold  function.  The  probability  expressions  from  NVThermlP 

are:’'’"' 

E  =  l.5l+0.2A(NJV,,)  and  =  iV,  (1- 0.585  (1-0.585  5 

l  +  CV/y^o) 

where  Nr  is  the  resolved  spatial  frequency,  Srh  and  Srv  are  the  spurious  responses  (horizontai  and 
verticai) 

NEI,  NETD  and  MRTD  (lab  MRTD)  remain  the  critical,  most  accurate  and  performance-telling 
metrics  and  the  current  model  now  provides  accurate  predictions  of  these  parameters. 
Probabilities  of  detection,  identification,  and  recognition  and  the  ranges  for  each  are  much  more 
varied  due  to  the  specific  target  shapes  and  aspects,  the  field  environment  (lighting  conditions, 
vibration,  display  glare)  and  background  clutter.  We  can  usually  assume  that  sensors  with 
similar  NETD’s  and  MRTD’s  will  provide  similar  target  range  estimations.  The  major  variability 
factors  then  are  the  observer,  background,  atmosphere  and  target  which  are  captured  to  some 
extent  in  the  range  probability  equations;  however,  these  rely  also  on  additional  empirical  values 
derived  from  field  observations  on  specific  targets  and  target  sets  under  specific  conditions. 

2.4  Observations/conclusions  from  manufacturer’s  data  and  model  results 
2.4.1  Relative  strengths  of  noise  sources 

Initial  looks  at  VOx  and  alpha-silicon  detectors  using  the  model  and  using  data  from  DRS  and 
L3  papers  indicate  that  both  detectors  are  limited  by  1/f  noise  at  the  high  bias  voltages  used  to 
maximize  the  response’'’""  ’'’'’".  The  a-Si  is  more  dominated  by  1/f  noise  but  it  is  also  operated 
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typically  at  4-5  volts  where  the  VOx  is  at  2-2.5  volts.  The  higher  resistance  of  a-Si  allows  for  full 
frame  time  integration  while  the  VOx  is  limited  to  a  bias  pulse  and  integrate  to  inhibit 
overheating  the  bolometer.  From  DRS  data  (17  micron  pixels  at  2.4  volts  bias),  after  1/f  noise, 
the  next  highest  noise  source  for  the  VOx  is  temperature  fluctuation  noise,  followed  by  readout 
noise,  then  Johnson  noise.  From  L3  data  (20  micron  pixels  at  5  volts  bias),  the  second 
strongest  noise  source  is  Johnson  noise,  then  readout  noise,  then  temperature  fluctuation 
noise.  While  the  absolute  noise  values  are  not  given  for  the  DRS  bolometer,  the  ratios  from  the 
chart  are  4.6,  3.7,  3.3,  2.8  for  1/f,  temp  fluctuation,  readout,  and  Johnson.  The  model  predicts 
similar  ratios  even  though  layer  thicknesses  and  device  structure  are  not  known  very  well.  For 
the  L3  a-Si  the  values  are  26, 12, 1 1 , 6.5  for  1/f,  Johnson,  readout,  and  temp  fluctuation.  This 
model  predicts  similar  ratios  here  also.  The  ratios  are  tabulated  in  Table  3. 


Noise  type 

DRS  data 

RR  model 

Noise  type 

L3  data 

RR  model 

1/f 

4 

4  (2e-13) 

1/f 

1.64 

1.64  (6e-12) 

johnson 

1.85 

2.38 

Temp  fluct 

1.32 

1.29 

Read 

1.69 

1.69 

Read 

1.18 

1.18 

Temp  fluct 

1 

1 

Johnson 

1 

1 

Table  3:  Relative  ratios  of  noise  from  data  packages  and  this  modei’s  predictions 


2.4.2  1/f  noise  detail 

The  level  of  1/f  noise  is  determined  in  the  model  by  the  value  of  the  Hooge  parameter  or  the  k 
factor  which  at  this  point  are  not  precisely  known  for  the  VOx  and  a-Si.  In  the  literature  a  typical 
Hooge  value  for  VOx  is  about  10'^^  cm^,  but  depending  on  processing  details  this  value  can  vary 
greatly.  We  can  extract  a  Hooge  parameter  value  or  k  factor  from  the  data  if  the  level  of  1/f 
noise  is  given.  In  the  above  table  the  1/f  noise  was  modeled  using  the  k  factor.  Agreement  with 
data  was  found  for  k’s  of  2e-13  for  VOx,  and  6e-12  for  a-silicon.  This  indicates  that  the  1/f  noise 
from  the  alpha-silicon  is  about  V®  or  about  8  times  higher;  however  this  does  not  seriously 
degrade  the  a-Si  NETD  since  the  alpha-silicon  noise  bandwidth  is  much  smaller  and  it  is 
operated  at  about  twice  the  bias  voltage.  Consequently  1/f  noise  based  NETD  limits  look  to  be 
very  similar.  1/f  noise  will  also  depend  on  the  layer  thickness  (volume  of  material)  so  this  must 
also  be  known  for  each  type  of  detector  to  be  able  to  predict  1/f  noise. 

2.4.3  Response  times  and  sensitivity  comparisons 

It  appears  from  the  limited  data  seen  so  far  that  sensitivity  (NETD)  is  comparable  for  the  VOx 
and  a-Si  based  bolometers.  It  may  be  that  the  higher  voltage,  TCR  and  response  available  to 
the  a-Si  compensates  for  a  higher  level  of  1/f  noise.  As  far  as  response  times,  the  VOx  may  be 
inherently  slower  due  to  the  requirement  for  an  absorbing  layer  and  an  underlying  substrate. 

The  alpha-silicon  can  be  grown  on  a  sacrificial  substrate  like  a  polymer,  and  doesn’t  require  an 
absorber  so  a  lower  heat  capacity  can  be  achieved.  For  the  smaller  pixel  sizes  being 
developed,  the  model  shows  that  the  leg  conductance  is  becoming  dominated  by  the 
metallization,  so  thinning  and  lengthening  of  the  legs  may  not  achieve  as  much  of  an 
improvement  in  conductance  as  expected.  None  of  these  conclusions  should  be  considered  set 
in  stone  though,  as  little  is  known  of  the  individual  manufacturers’  specific  capabilities  in 
controlling  these  device  parameters.  The  Table  4  below  shows  model  results  for  pixels  of  VOx 
and  a-Si  for  identical  C  and  G  values  (first  3  columns),  then  for  reduced  C  and  G  (last  two 
columns).  The  NETD  predicted  for  a  typical  VOx  at  2.5V  is  40mK  and  is  1/f  noise  limited.  The 
alpha-silicon  with  the  higher  1/f  k  value  has  an  NETD  of  over  lOOmK.  Reducing  the  1/f  noise  k 
value  to  that  used  for  the  VOx  gives  a  total  NETD  for  the  a-Si  bolometer  at  5V  (column  3)  of 
57mK,  this  time  limited  by  the  Johnson  noise.  A  twofold  reduction  in  size  (C  and  G)  gives  total 
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a-Si  NETD’s  of  55mK  and  30mK.  From  these  initial  simulations  it  appears  that  as  the  data  has 
shown,  the  performance  of  these  two  detectors  will  be  similar  and  final  performance  will 
probably  be  determined  by  manufacturing  subtleties  and  readout  electronics.  Spatial  noise  has 
not  been  mentioned  and  this  may  also  become  a  factor  in  overall  performance.  Nonuniformity 
correction  (calibration)  must  be  sufficient  to  limit  spatial  NETD  to  a  level  significantly  below  the 
temporal  NETD. 


Table  4:  NETD’s  predicted  from  model  for  like-detectors  of  VOx  and  a-Si  for  different  levels  of  1/f 

noise. 

2.5  Simulation  of  microbolometer  sensors  for  aerial  collision  avoidance 

In  these  examples  we  look  at  a  wide  angle  FOV  (from  20  to  40  degrees)  with  f/1  optics  for  small 
aerial  object  detection.  The  frame  rate  is  30Hz,  the  spectral  band  is  8-12  microns,  the  baseline 
NETD  is  40mK,  and  the  optical  efficiency  is  70%.  The  intended  target  is  at  altitudes  from  100 
feet  to  1 0,000  feet  and  the  sensor  target  path  is  horizontal  (for  sky  radiance  and  atmospheric 
path  transmittance  calculations).  Atmospheric  transmittance  and  radiance  data  from  many 
MODTRAN  runs  over  the  8-12  micron  spectral  range  were  compiled  and  a  model  in  excel  was 
developed  from  these  data  tables’'’'’'".(18)  Horizontal  path  sky  radiances  were  compiled  for 
these  altitudes  and  were  compared  to  blackbody  curve  outputs  to  assign  an  effective  blackbody 
temperature  to  the  path  inband  radiance  values  from  MODTRAN.  These  effective 
temperatures  vs.  altitude  are  shown  in  Figure  16a  which  were  derived  from  MODTRAN  model 
runs.  Note  that  the  actual  temperatures  at  the  altitudes  are  higher  than  the  effective 
temperature.  The  effective  temperature  takes  into  account  the  path  emittance  from  the  sensor 
to  space  which  is  less  than  unity.  For  example,  the  MLS  sea  level  temperature  is  25C  or  77F  or 
298K,  but  the  horizontal  path  radiance  due  to  absorption  in  the  8-12  micron  band  is  only  80%  of 
the  ideal  spectral  sum  radiance  of  a  perfect  298K  blackbody  radiator.  The  80%  emissivity  298K 
radiator  is  equivalent  to  a  100%  emissivity  285K  radiator.  This  285K  is  the  effective 
temperature  used  in  the  model  to  calculate  inband  path  radiance.  From  MODTRAN  runs  at 
each  altitude  we  get  the  effective  temperatures  that  are  used  in  the  excel  model  to  approximate 
horizontal  path  radiances.  For  the  path  transmittances,  MODTRAN  runs  at  various  altitudes 
and  path  distances  were  used  to  find  absorption  coefficients  as  functions  of  altitude  and  path 
length.  These  are  shown  in.  These  expressions  are  used  to  calculate  the  SNR’s  for  targets  of 
defined  areas  and  temperatures,  altitudes  and  ranges. 
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Figure  16:  Effective  temperatures  of  horizontai  path  vs.  aititude  (ieft)  and  8-1 2um  transmittances 
vs.  range  for  aititudes  from  sea  ievei  to  10km  (right) 

Sensors  and  scenarios  were  investigated  to  predict  SNR’s  from  aircraft  at  various  temperatures 
and  areas,  and  at  various  ranges  and  altitudes.  A  few  of  the  common  bolometer  camera  FPA 
formats  and  FOV’s  were  simulated  and  the  plots  in  Figure  17  were  generated  from  these  excel 
model  runs. 

For  SNR  calculations  for  subpixel  targets,  we  use  the  NEP  or  the  t/no/se/  Response  ratio.  The 
SNR  is: 

SNR  =  [P,^^-P^^„^,^]/NEP 

where  Ptar  is  the  target  energy  on  the  sensor  optics  (watts)  and  Pscnobs  is  the  contribution  from  the  path 
radiance  that  is  obscured  by  the  target. 

The  response  is  given  as: 

^  _  ^tcr^b 

^  G  (R,+R,f 

The  cases  modeled  were: 

Case#1:  40  deg  FOV,  0.5”  optics,  28um  320x240  FPA,  f/1,  target  deltaT  =  30F,  target  area  = 
4m^,  for  altitudes  of  1 00ft.  to  20kft. 

Case#2:  20  deg  FOV,  0.5”  optics,  28um  160x120  FPA,  f/1,  target  deltaT  =  5F,  15F  and  30F, 
target  area  =  2m^,  for  1 00ft. 

Case#3:  25  deg  FOV,  1 .0”  optics,  1 7um  640x480  FPA,  f/1 ,  target  deltaT  =  1 30F,  target  area  = 

1  m^,  for  1 0,  20  and  30kft. 

These  SNR  plots  are  as  accurate  as  the  known  values  for  the  camera  NETD  and  the  optical 
efficiency.  Reported  values  for  f/1  NETD’s  range  are  in  the  20mK  to  50mK  region  for  most  of 
these  uncooled  sensors.  The  general  trend  is  a  decrease  in  NETD  with  pixel  size.  The  NETD 
was  taken  to  be  40mK  for  all  cases  and  the  optical  efficiency  (which  includes  the  optics 
transmittance)  was  70%.  If  the  actual  NETD  and  optical  efficiency  are  different,  the  plots  will 
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move  slightly  up  or  down  linearly  with  the  NETD  and  optical  efficiency  ratios.  That  is,  if  the 
optical  efficiency  is  60%,  the  SNR  plot  should  be  lowered  by  0.6/0.7. 

The  signal  was  calculated  as  the  difference  in  level  (voltage)  of  the  pixel  containing  the  target  to 
a  nearby  pixel  that  is  observing  only  sky  radiance.  The  accuracy  was  checked  by  adding  a 
285K  target  in  the  pixel  IFOV  (target  being  the  same  temperature  as  the  effective  sky 
temperature).  The  signal  here  was  seen  to  be  zero.  As  an  additional  check,  the  target  was 
made  to  fill  the  IFOV  and  set  at  285. 04K.  Flere  the  delta  is  equal  to  the  NETD  of  the  uncooled 
sensor.  The  signal  to  noise  was  exactly  one  when  the  effect  of  the  atmospheric  transmittance 
was  removed. 

The  sky  radiance  was  modeled  from  MODTRAN  runs  for  horizontal  paths  at  the  specified 
altitudes.  The  inband  target  radiance  was  calculated  from  a  blackbody  emitter  at  the  specified 
area  and  temperature,  diminished  by  the  atmospheric  path  transmittance.  The  path  radiance 
(target  to  sensor  path)  was  added  back  into  the  expression  so  that  a  target  at  ambient 
temperature  produces  zero  signal.  The  signal  was  diminished  by  the  optical  efficiency  as  in  this 
case,  30%  of  the  target  signal  will  be  collected  by  the  target  pixel.  In  the  bolometer  model,  the 
delta  signal  produces  a  temperature  difference  and  resistance  change  in  the  bolometer  which  is 
sensed  as  a  voltage  difference.  This  voltage  difference  divided  by  the  total  noise  voltage  is  the 
SNR.  The  target  was  moved  from  1km  to  up  to  50km  in  horizontal  range  and  SNR’s  were 
tabulated  for  all  cases. 

The  top  left  plot  in  Figure  17  for  case#1  shows  SNR’s  for  the  40  deg  FOV  320x240  sensor  and 
the  +30F  4m^  target.  As  expected,  SNR’s  are  shown  to  be  higher  for  the  lower  altitudes 
tracking  with  the  atmospheric  transmittance.  The  SNR  =  6  level  for  this  target  is  in  the  6  to  14 
km  range  for  these  altitudes.  The  +30F  target  is  defined  as  30F  above  the  sea  level 
temperature  (77F)  or  107F  and  remains  107F  at  all  altitudes.  Note  that  at  the  lower  two 
altitudes,  the  transmittance  diminishes  the  signal  much  faster.  The  higher  altitudes  (above  3000 
ft)  allow  for  about  twice  the  SNR6  range. 

Case#2  (upper  right  plot  Figure  17)  for  the  160x120  20  degree  FOV  sensor  shows  improvement 
in  SNR  with  target  temperature  delta.  All  SNR  of  6  levels  occur  within  the  3.7  to  4.7  km  range 
for  the  +5F  to  +30F  2m^  targets. 

In  Case#3  for  the  25  degree  FOV  640x480  sensor,  the  SNR  of  6  levels  from  the  1m^  130F  delta 
target  are  in  the  27  to  35  km  range.  The  plot  at  lOkft  shows  more  attenuation  of  signal  than  the 
plots  at  20kft  and  30kft  which  are  nearly  identical. 

NETD’s  in  current  sensors  will  probably  be  close  to  the  estimated  40mK  for  the  small  pixels, 
where  for  the  larger  pixels,  NETD’s  may  be  as  low  as  30mK.  Vendor  data  has  shown  the  best 
FPA’s  can  achieve  NETD’s  lOmK  or  so  better  than  these  values;  however,  average  NETD’s 
over  a  decent  sample  of  detector  arrays  are  more  in  the  30-50mK  range.  From  the  limited  data 
acquired  from  the  vendors  for  this  study,  it  remains  unclear  what  the  limiting  noise  mechanism 
will  be  for  the  alpha-silicon  and  the  vanadium  oxide  detectors.  1/f  noise  may  be  the  limiting 
noise  source  for  both  detectors,  especially  at  the  higher  bias  voltages;  however,  hard  numbers 
(k-values  or  Flooge  parameters)  were  not  available.  If  the  best  detectors  are  used  (lower  NETD 
values),  the  case  #1  and  #2  ranges  may  be  increase  somewhat.  Cooling  the  FPA  to  220K  may 
also  improve  NETD  by  up  to  a  factor  of  two  according  to  some  L3  data  on  their  alpha-silicon. 

We  have  not  seen  any  data  on  vanadium  oxide  at  lower  temperatures  for  comparison. 
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Figure  17:  SNR’s  vs.  target  range  for  case  1  and  2  sensor  configurations 


Figure  18:  SNR’s  vs.  range  for  case  3  Figure  19:  SNR’s  vs.  range  with  NETD  varied 

from  20  mK  to  50  mK 

In  Figure  19  we  reproduce  the  results  with  some  differences  in  NETD  to  show  its  effect  on  SNR 
and  range.  The  baseline  NETD  was  40mK  and  other  typical  and  possible  NETD’s  of  20,  30, 
and  50mK  produce  the  plots  shown  in  Figure  19.  In  general  we  see  increases  in  range  with 
reduction  in  NETD.  For  the  SNR=6  criteria,  the  ranges  for  50,  40,  30  and  20mK  NETD’s  are  5, 
5.6,  6.5  and  8  km. 

Uncertainty  in  NETD,  in  optical  efficiency  (a  function  of  the  optical  spot  size  and  detector  area), 
and  variations  in  atmospheric  transmittance  and  turbulence  will  all  add  uncertainty  to  these 
range  estimations.  For  the  lower  altitudes,  the  transmittance  will  vary  much  more  significantly 
with  different  levels  of  humidity  and  aerosol.  Currently,  three  atmospheres  have  been 
generated  for  use  in  calculating  SNR,  the  U.S.  standard  1976  model  with  23  km  visibility  (rural), 
the  mid-latitude  summer  with  23km  visibility  and  then  with  5  km  visibility  (both  rural).  The  8-12 
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micron  band  transmittances  are  shown  in  Figure  20  for  these  three  choices.  These  charts  were 
generated  from  MODTRAN  runs  for  horizontal  paths.  The  MODTRAN  data  were  fit  with 
exp(-a*range*b)  expressions  which  give  very  accurate  fits  to  the  data  from  1  to  50  km  path 
lengths. 

The  effect  on  SNR  from  atmosphere  variability  is  shown  in  Figure  21  for  the  case  #2.  A  +30F 
target  gives  SNR’s  vs.  range  as  follows  for  altitudes  of  0  to  3  km  for  three  specific  atmosphere 
prescriptions.  The  US  1976  Standard  atmosphere  with  23  km  visibility  (aerosols)  gives  the 
highest  SNR’s.  The  second  best  is  the  midlatitude  summer  model  with  23  km  visibility  and  the 
third  best  is  the  midlatitude  summer  with  5  km  visibility.  For  the  sea  level  altitudes,  the  SNR=6 
range  for  the  three  atmospheres  are  from  about  4  km  to  5.5  km.  Going  to  a  1  km  altitude 
improves  the  range  to  5  to  7  km.  The  SNR=6  range  for  the  US  Standard  atmosphere  is  about  9 
km  for  the  3  km  horizontal  path. 


Figure  20:  Three  atmospheres  currently  simulated  in  program 
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snr  vs.  range  for  3  atmospheres 
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Figure  21 :  Case  #2  with  a  +30F  target  and  at  aititudes  from  0 
to  3  km  for  the  1976  US  standard  23km  vis,  the  MLS  23km  vis 
and  the  MLS  5km  vis  atmospheres. 

2.6  Some  unique  attributes  of  the  modei 

The  model  consists  of  some  unique  characteristics  that  were  developed  to  better  characterize 
microbolometers  and  uncooled  cameras.  Some  of  these  attributes  are: 

1 .  Detailed  equations  for  NETD  and  NEP  from  first  principles:  bolometer  geometries  and  layer 
characteristics. 

2.  Radiometric  expression  derived  for  background  solid  angle  gives  more  accurate  calculation  of 
NETD  for  low  f-numbers. 

3.  NETD  expressions  allow  for  centrally  obscured  optics  (reflective),  as  do  the  diffraction  spot 
size  calculation  (EOD  for  diffraction  limited  optics  which  is  used  in  the  NEP  and  NEI 
calculations). 

4.  Response  is  calculated  for  all  contributions  from  the  heat  balance  equation,  not  just  the  usual 
approximation  using  the  leg  thermal  conductance. 

5.  Creation  of  simulated  bar  target  imaging  sequences  that  when  observed  by  the  human  eye 
give  MRTD’s  that  are  in  agreement  with  NVTherm  MRTD(f)  results. 

6.  Full  use  and  validation  of  model  to  NVTherm  equations  and  outputs  for  MTF,  MRTD  and 
probabilities  of  detection,  identification  and  recognition. 

7.  Model  provides  a  seamless  transition  from  subpixel  region  (point  object  images  and 
performance  predictions,  NEP  and  SNR)  to  resolved  region  (extended  object  images  and 
performance  predictions,  NETD  and  MRTD). 

The  differences  in  NETD  due  to  the  scene  solid  angle  expression  used  are  shown  in  figure  19. 
Expressions  found  in  the  literature  were  either  of  the  form  4F^  or  4F^+1 .  The  first  overestimates 
the  solid  angle,  the  second  underestimates  it.  All  expressions  converged  for  the  larger 
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f/numbers,  but  significantly  differ  from  f/1 .5  on  down.  The  expression  used  in  the  model  (middle 
curve)  gives  an  accurate  value  for  the  area  of  a  sphere  subtended  by  a  solid  angle  phi  for  all  f- 
numbers. 


Figure  22:  solid  angle  of  scene  (phi/pi)  gives  more  accurate  NETD  at  f/1  and  lower 

2.7  Summary  and  discussion 

A  microbolometer  model  has  been  developed  and  validated  that  predicts  accurate  NETD  values 
for  specified  bolometer  materials,  geometries  and  electronics.  The  model  was  implemented  in 
excel  and  in  matlab  to  provide  detailed  single  pixel  predictions  and  realistic  image  sequences 
complete  with  nonuniformities  and  3-D  noise.  The  model  was  extended  to  predict  MRTD’s  and 
probabilities  in  accordance  with  and  validated  to  the  Night  Vision  models  (NVTherm2002  and 
NVThermlP).  Both  NETD’s  predicted  by  the  model  and  MRTD  plots  generated  by  the  model 
are  in  close  agreement  with  other  model  results  from  literature  and  to  NVTherm  outputs  and 
simulated  visual  bar  target  observations.  The  model  was  applied  to  the  mission  of  collision 
avoidance  to  predict  SNR’s  and  ranges  for  a  variety  of  scenarios.  Further  efforts  in  validation 
and  model  development  (in  order  to  fully  predict  and  characterize  performance)  rely  on 
discerning  the  practical  limits  in  bolometer  pixel  manufacturing  and  the  bolometer  materials 
semiconductor  characteristics  such  as  1/f  noise,  TCR,  resistance  and  absorbance.  Knowledge 
of  the  vendors’  electronics  readout  and  noise  reduction  schemes  and  effectiveness  will  also  be 
required  to  determine  if  electronics  noise  and  spatial  noise  can  be  kept  below  the  total  of  the 
other  four  types  of  temporal  noise  (1/f,  Johnson,  thermal  fluctuation,  and  background 
fluctuation)  for  future  designs.  The  model  as  it  is  predicts  the  major  noise  components  in 
correct  proportion  for  both  the  alpha-silicon  and  vanadium  oxide  microbolometers 

Some  observations  here  that  look  to  be  typical  of  f/1  uncooled  microbolometer  sensors  are  that 
both  types  of  detectors  (VOx  and  a-Si)  are  limited  by  1/f  noise  at  the  higher  bias  voltages; 
otherwise,  higher  biases  would  continue  to  improve  performance.  In  general,  the  a-Si 
bolometers  exhibit  slightly  faster  responses,  but  also  may  have  higher  inherent  1/f  noise.  MRTD 
predictions  from  my  matlab  images,  NVTherm  runs,  and  from  the  various  equations  show  that 
for  a  large  part  of  the  spatial  frequency  range,  the  MRTD  remains  less  than  the  NETD,  up  to  a 
factor  of  two  to  three  times  less.  This  is  a  direct  result  of  the  eye  integration  which  sums  a  few 
frames  (signal)  while  rejecting  some  of  the  temporal  noise  which  provides  a  square  root  of  the 
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number  of  frames  integrated  improvement  in  SNR  and  MRTD.  The  eye  CTF(f)  and  the  system 
MTF(f)  functions  are  the  other  variables  used  in  the  MRTD  equation.  Eye  CTF  is  dependent  on 
ambient  illumination  and  display  characteristics  and  the  system  MTF  in  the  model  is  comprised 
of  diffraction  effects,  detector  sampling  effects,  and  spurious  response  or  aliasing. 

3  Hardware  and  Software 

To  accomplish  the  goals  of  the  program  COTS  technology  was  extensively  leveraged.  Off  the 
shelf  microbolometer  cameras  and  existing  data  collection  software  was  modified  to  meet  the 
needs  of  the  program. 

3.1  Data  Collection  Software 

StreamsS  (www.ioindustries.com)  a  Windows  based  multistream  data  collection  software.  It 
allows  the  simultaneous  collection  of  data  streams  in  multiple  formats  to  multiple  devices 
(Figure  23).  This  video  transmission  formats  used  in  this  program  was  analog,  LVDS, 
CameraLink  and  GiGE.  It  is  deeply  integrated  with  Windows  via  C++  and  has  open  source 
PYTFION  scripting  capability  www.pvthon.orq.  lO  Industries  was  hired  to  code  special  drivers 
for  the  GiGE  interface  and  IMU  integration.  The  data  was  exported  in  a  RAW  format  for  use 
with  MatLab  (UConn)  and  MatCad  and  Imaged  (MilSys).  ImaaeJ  is  a  Java  based  open  source 
image  processing  program  supported  by  the  NIH. 


Figure  23:  StreamsS  screen  capture 

Several  custom  models  were  created  to  support  hardware  setup  and  GigE  driver  integration 
(Figure  24).  The  L-3  hardware  setup  was  via  a  USB  port.  The  GigE  was  controlled  through  an 
iPort  driver  developed  by  Pleora.  The  IMU  (inertial  measurement  unit)  was  programmed  via  the 
iPort  through  a  UART  to  the  FPGA  (See  3.4). 
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Figure  24:  Modules  to  developed  to  control  the  camera  and  IMU. 


3.2  COTS  Microbolometers 

The  two  core  technologies,  Vox  and  a-Si,  (Table  5)  for  microbolometers  were  extensively 
studied  and  reviewed  for  the  program.  The  BAE  camera  uses  the  SCC500  core  with  a  third 
party  CameraLink  interface  and  lens  from  OPHIR.  The  L-3  4550AS  came  complete  with  lens, 
analog  output,  and  a  16bit  digital  OEM  port.  MilSys  designed  for  it  an  interface  board  for  the 
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OEM  port  for  both  the  LVDS  and  GigE  output  (See  3.4).  Unsuccessful  attempts  were  made  to 
purchase  a  DRS  640  X  480  Vox  microbolometer.  Indigo  (FLIP)  also  sells  a  similar  package,  but 
was  not  contacted  due  to  the  potential  non-disclosure  issues.  Raytheon  has  a  very  small 
package  (Figure  25),  however  due  to  their  commitment  to  meet  DoD  sales,  at  the  time  no  cores 
were  available  for  testing. 


I 

Manufacturer 

BAE 

L-3 

Optics 

^1” 

^0.5” 

Pixels 

640  X  480 

320  X  240 

FOV 

28° 

54° 

Pixel  Size 

25).im 

30).im 

Technology 

VOX 

a-Silicon 

Range 

8-12|im 

7-14)im 

Sensitivity 

<50mK 

<50mK 

Table  5:  Comparison  of  the  specifications  of  the  two(2) 
main  MB  technoiogies. 

As  part  of  the  development  process  the  noise  and  S/N  ratio  of  the  cameras  (both  VOX  and  a- 
Silicon)  was  measured.  Several  macros  were  written  in  Imaged  to  process  the  videos. 


Figure  25:  Raytheon  microbolometer 
320  X  240,  25um  with  eiectronics 


Figure  26:  STD  and  average  of  the  L-3  microboiometer.  Note  the  red  circie  is  a  defect  in  the  FPA. 

The  images  above  (Figure  26)  are  from  the  a-Silicon  viewing  a  wall.  The  above  left  image  is  a 
standard  deviation  (n=100)  of  the  still  image  (AVG  n=100)  on  the  right.  The  STDEV  is  uniform 
over  the  entire  image.  The  population  average  of  the  STDEV  is  approx  1 1 .5  counts  out  of 
1 6bits.  Note  the  red  circle;  it  is  a  problem  with  the  camera  developed  during  the  first  two 
months.  Interestingly  it  does  not  affect  the  measurements.  It  was  returned  to  the  factory  for 
repair. 
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The  images  above  are  from  the  BAE  Vox  viewing  the  same  wall  as  the  a-Silicon.  The  above  left 
image  is  a  standard  deviation  (n=100)  of  the  still  image  (AVG  n=100)  on  the  right.  The  STDEV 
indicates  some  fixed  pattern  noise  within  the  image.  The  population  average  of  the  STDEV  is 
approx  1 .5  out  of  14bits.  The  results  indicate  that  both  cameras  have  a  S/N  12bits  (4096:1). 


The  cameras  were  used  in  three  configurations:  BAE  and  L-3  parallel  looking  at  the  same 
scene  (Figure  27),  L-3  and  L-3  side  by  side  for  tracking  across  the  FOV  (Figure  28),  and  L-3 
and  L-3  both  parallel  (not  shown). 


Figure  27:  Camera  system  used  to  collect  initial  Figure  28:  Dual  L-3  Camera  used  in  later  side 
data  of  Ultralight.  Both  BAE  and  L3  are  shown.  by  side  data  collections. 


3.3  GPS  Data  Logger 

To  track  the  position  of  the  aircraft  relative  to  other  aircraft  or  the  ground,  portable  GPS  data 
loggers  were  purchased  from  Landairsea  (Figure  29).  This  model  was  chosen  for  its  accuracy, 
2.5meters  and  its  small  size.  The  update  rate  is  only  1  FIz,  which  corresponds  to  88ft/s 
@60mph.  The  factory  was  contacted  about  increasing  the  rate,  however,  it  is  not  possible  with 
this  type  of  technology  at  the  price  point.  In  general,  this  resolution  was  deemed  accurate 
enough  for  the  tasks  required.  At  the  start  of  each  data  collection,  the  logger  was  located  at  the 
camera  mount  and  then  placed  in  the  pilots  pocket  or  on  board  the  aircraft.  The  results  were 
later  exported  into  their  customer  software,  a  text  file  or  Google  Earth  for  post  processing. 
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Figure  29:  LandAirSea  GPS  data 
logger. 


Figure  30:  An  example  of  the  data  export  from  the  GPS  tracker  into  Google  Earth. 
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3.4  Aircraft 

Two  types  of  light  aircraft  were  used  for  data  collection  Piper  (Figure  31  &  Figure  32)  and  Kolb 
Ultralight  (Figure  34).  The  original  work  by  Priest  et  al  used  a  Cessna  as  its  standard  target; 
however,  a  Piper  is  a  close  match  (Table  6)  and  was  a  more  available  aircraft.  Most  of  the  data 
collections,  ground-air  and  air-air  used  the  Kolb  Ultralight.  The  pilot  was  very  cooperative  and 
he  flew  out  of  the  uncontrolled  airport  at  Alloway  NJ  (See  3.6). 


Figure  31 :  Piper  aircraft  used  for  data  coilections. 


Figure  32:  Piper  from  website.  Figure  33:  Cessna  from  website. 


Piper 

Cessna 

Maximum  Take-off 
Weight 

2,750  lbs/1 ,247  kg 

2450  lbs/1 11 1kg 

Useful  Load 

960  lbs/437  kg 

758  lbs/343kg 

Length 

24.7  ft/7.5  m 

27'2"/8.3m 

Fleight 

7.9  ft/2.4  m 

8’11”ft/2.7m 

Wing  Area 

170ft''/15.79  m2 

174ft''/16.2  m"" 

Wing  Span 

35.4  ft/1 0.8  m 

36'1 71 1  m 

MAXIMUM  SPEED 

145kts/269  km/h 

123kts/228  kph 

CRUISING  SPEEDS 

137kts/254  km/h 

75%  @8kft 

122kts/227  kph 

80%  @8kft 

Table  6:  Specifications  for  Piper  and  Cessna. 

31 


Figure  34:  Kolb  FireStar  Ultralight  33ft  wing  span  cruising  speed  60nnph. 


The  Ultralight  is  constructed  of  tubular  aluminum  and  fabric  wings.  The  engine  is  a  multistroke 
“lawnmower”  type  with  a  large  muffler.  The  engine  is  pointed  towards  the  rear  of  the  aircraft. 
The  pilot  indicated  that  the  muffler  during  operation  approaches  1200F  (Figure  35).  At  range, 
the  wings  provide  some  thermal  contrast  with  the  background  sky,  however  the  primary 
background  contrast  or  hot  spot  is  provided  by  the  muffler. 


Figure  35:  LWIR  image  of  the  Ultralight.  Muffler  runs  at 
approximately  1200°F. 


3.5  Camera  hardware  development 

The  BAE  camera  was  purchased  from  a  third  party  integrator  which  provided  a  CameraLink 
interface  providing  a  full  digital  data  collection.  However,  the  L3  camera  core  was  only  available 
with  analog  output  or  OEM  16bit  digital  port.  Custom  hardware  is  required  to  use  the  OEM  port. 
In  the  first  phase,  a  OEM  to  LVDS  interface  was  designed  (Figure  36-Figure  38).  The  LVDS 
interface  uses  a  0.5”  diameter  stiff  cable.  This  cable  was  determined  to  be  not  compatible  with 
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the  goal  of  developing  a  small  data  logging  system  that  could  be  flown  in  multiple  platforms, 
ranging  from  a  small  UAV  to  a  larger  manned  aircraft. 


Figure  36:  3D  model  of  LVDS  circuit  board  Figure  37:  Silk  screen  of  circuit  board, 
with  L-3  AS4500. 


LVDS 

Cable 


Figure  38:  Assembled  camera  and  cable.  Figure  39:  L3  Core  with  LVDS  circuit  board. 

The  second  stage  of  development  was  to  integrate  IMU  with  a  GigE  interface.  The  GigE 
interface  was  chosen  because  of  its  high  bandwidth  and  small  cable  size  (Figure  40-Figure  42). 
The  IMU  was  an  integrated  module  produced  by  Analog  Devices  AD16355.  It  is  based  on  their 
iMEMs  process.  Two  cameras  were  completed  for  use  in  air-air  testing. 


Figure  40:  3D  model  of  GiGE  and  IMU  with  L-3  Figure  41 :  View  of  the  completed  camera. 
AS4500. 
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Figure  42:  View  of  the  completed  camera. 

To  collect  air-air  data  an  ITX  form  factor  PC  (Figure  44)  was  chosen  due  to  its  size,  power 
requirements,  and  dual  GiGE  ports.  A  small  ruggedized  80GB  disk  drive  was  install.  A  portable 
Lead  Acetate  battery  and  charger  were  installed  in  the  Kolb  Ultralight  behind  the  pilots  seat 
(Figure  43).  The  camera  was  installed  on  a  removable  mount  on  the  nose  (Figure  45).  The 
mount  was  isolated  from  aircraft  with  vibration  mounts.  The  engine  is  also  isolated  from  the 
airframe  wit  vibration  mounts.  The  pilot  indicates  that  the  aircraft  does  not  vibrate  due  to  the 
engine. 
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Figure  43:  Installation  of  the  computer, 
battery,  and  DC  charger. 


Figure  44:  Mini-ITX  computer. 


Figure  45:  View  of  the  completed  camera  installed  in  the  Ultralight. 


3.6  Airports 

All  of  the  data  collections  took  place  at  two  airports,  Pottstown,  PA  (Figure  46-Figure  48)  and 
Alloway  (Figure  49  &  Figure  50),  NJ,  during  2007-2008.  Pottstown  is  a  public  ATC  local 
municipal  airport.  Alloway  is  a  private  uncontrolled  airport.  Due  to  the  flexibility  and  lack  of 
restrictions,  Alloway  airport  was  the  primary  chose  for  data  collections. 


Figure  46:  Ariel  View  with  flight  path  of  the  Pottstown  Airport,  PA. 
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Figure  47:  Dual  camera  setup  at  Pottstown 
Airport. 


Figure  48:  Ariel  View  of  the  Pottstown 
Airport,  PA. 


Figure  49:  Ariel  View  of  the  Alloway  Airport,  Figure  50:  Ground  level  view  of  the  Alloway 
NJ.  Airport. 


4  Representative  Data  Collections 

The  following  is  a  representative  sampling  of  the  ground-air  data  collections  and  their  range 
estimates  (Figure  51 -Figure  54  &  Table  7). 
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Figure  51 :  BAE  (left)  and  L-3  (right)  collected  at  the  Alloway  Airport  using  camera  configuration  in 
Figure  27. 


Figure  52:  L-3  both  parallel  (Figure  28)  collected  at  Pottstown  Airport. 


Figure  53:  L-3  side  by  side  (Figure  28)  coliected  at  Pottstown  Airport. 


Figure  54:  Frame  showing  Piper  crossing  from  one  camera  FOV  to  another  iater  in  the  data 
coilection  of  Figure  53. 


On  Oct  31  both  the  BAE  and  L-3  cameras  were  operational  and  using  the  StreamsS  software 
the  video  was  collected  synchronously.  The  pilot  was  instructed  to  pass  as  close  as 
comfortable  above  the  camera  tripod  and  fly  straight  for  3  miles  climbing  to  1000ft.  The  pilot 
had  a  GPS  data  logger  in  his  pocket.  Actuality  the  pilot  trended  to  the  right  an  flew  out  of  the 
FOV  of  the  cameras  on  two  of  the  three  passes.  Based  on  the  Oct  30  video,  on  Nov  28  the  pilot 
was  instructed  to  fly  to  3000ft  with  the  thought  being  that  the  background  would  be  clearer  and 
lower  in  temperature  thereby  improving  the  range  performance  of  the  camera.  The  pilot  few 
straight  however,  he  reached  the  3000ft  at  3  miles.  Review  of  the  data  indicated  that  he  was  at 
approximately  the  same  altitude  at  the  limits  of  visibility  of  the  camera. 

Start  -  First  moment  that  the  aircraft  is  visible  in  the  scene 

End  -  Last  moment  that  I  can  see  the  aircraft.  Aircraft  is  at  one  pixel  and  the  Scene  is  zoomed 
on  the  monitor 

Comments  -  The  distance  was  determined  by  aligning  the  Start/End  Times  with  the  GPS  data. 
Note: 

1 )  When  in  black  and  white,  StreamsS  displays  256  levels  of  gray  the  window  and  level 
need  to  be  adjusted  to  try  to  enhance  the  visibility  of  the  single  pixel. 

2)  The  BAE  camera  has  4X  large  lens. 


Camera 

Start 

End 

Comments 

Scene  2 

BAE 

9:52:52 

9:54:40 

Dist  -1 .9m(3km) 

L-3 

9:52:52 

9:54:40 

Dist  -1 .9m(3km) 

BAE 

9:52:52 

9:55:14 

Dist  -  2.6m(4.1km) 

Note:  1  prematurely  cut  off  some  of  this  scene  data  during  editing 
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GPS  Data 

Scene2.txt 

AVG  Speed  63.6mph 

Flew  out  of  FOV 

Altitude  -  879ft 

Scene  3 

BAE 

10:13:55 

10:14:41 

Dist  -  0.9km(1 .5km) 

L-3 

10:13:55 

10:14:45 

Dist  -  0.9km(1 .5km) 

GPS  Data 

Scene3.txt 

AVG  Speed  60.7mph 

Flew  out  of  FOV 

354ft 

Scene  4 

BAE 

10:24:31 

10:27:36 

Dist  -  3.1m(5km) 

L-3 

10:24:29 

10:26:31 

Dist  -  2.0m(3.2km) 

GPS  Data 

Scene  4.txt 

AVG  Speed  60mph 

Altitude  958ft 

The  short  ranging  distance  for 
the  L-3  mimics  scene  2. 

Table  7:  Slant  Range  estimates  of  ground-air  data. 

Figure  55  is  the  initial  test  flight  of  the  hardware  developed  above  (Figure  42-Figure  45).  A 
second  Kolb  Ultralight  was  used  as  the  test  aircraft.  The  hardware  and  software  performed  as 
expected  and  both  video  and  IMU  data  was  collected.  The  second  pilot  was  not  expected  at  the 
initial  test  flight  and  the  GPS  dataloggers  (Figure  29)  were  not  brought  to  Alloway  Airport.  On 
the  inauguration  flight,  the  second  test  pilot  let  his  GPS  on  the  wing  of  the  aircraft  and  it  damage 
the  flap  and  the  propeller.  Before  the  aircraft  could  be  repaired  or  another  pilot  and  aircraft 
arranged  time  ran  out  of  this  program. 


Figure  55:  Initial  test  flight  of  the  IMU  for  Air-Air 
data  collection. 


The  IMU  collects  data  along  six  axis,  three  acceleration  and  three  angular  velocity.  Figure  55 
plots  the  absolute  roll  angular  velocity.  The  blue  line  represents  both  the  absolute  translation  of 
the  aircraft  in  degrees.  The  red  line  represents  the  translation  of  aircraft  due  to  fast  vibrations 
as  one  might  expect  due  to  the  motor  (2-5Hz).  The  goal  of  adding  the  IMU  to  the  camera  was 
required  to  stabilize  the  image  due  to  vibrations  not  physical  translations  of  the  aircraft.  The 
track  processor  would  use  the  INS  from  the  aircraft  to  correct  for  translations. 
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5  Algorithm  Development 

Dr.  Bar-Shalom  and  Dr.  Willet  are  world  renowned  researchers  in  the  area  of  small  target 
detectors  and  target  trackers,  along  with  Richard  Osborn  (grad  student),  they  were  hired  to 
develop  both  a  small  target  detector  and  a  tracker  for  the  PCAS  program.  Based  upon  their 
experience,  the  attitude  was  that  the  technology  tasks  have  been  solved,  and  that  what  was 
needed  was  a  unique  implementation  (i.e.  nothing  needs  to  be  invented).  (Due  to  the  software 
used  to  create  their  report,  it  is  included  here  as  a  static  reference.  The  data  provided  by  MilSys 
for  their  development  work  was  ground-air  only.)  As  discussed  below,  technical  difficulties 
arose  and  time  ran  out  on  the  present  contract  before  air-air  data  with  inertial  measurements 
could  be  provided.  Their  small  target  detector  is  based  upon  single  frame  processing  with  a 
measurement  window  ranging  from  50  X  50  to  20  X  20  pixels. 

5.1  UConn  Report 
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Passive  Collision  Avoidance  System,  Final 

Report 

R.  W.  Osborne,  TTI,  Y.  Bar-Shalom,  P.  Willett 
September  2,  2008 


1  Introduction 

Commercial  airliners  are  currently  equipped  with  systems  which  will  warn 
when  other  aircraft  are  within  an  unsafe  range  in  order  to  mitigate  the 
chances  of  a  midair  collision.  Currently,  the  unmanned  aerial  vehicles  (UAV), 
used  for  military  operations,  do  not  have  a  system  which  will  warn  of  impend¬ 
ing  collisions  with  other  airborne  targets.  Such  a  system  should  be  able  to 
track  uncooperative  targets,  and  therefore  must  not  rely  on  communications 
between  aircraft.  In  this  case,  the  sensors  used  for  tracking  will  be  imaging 
sensors  mounted  on  the  airframe  in  order  to  observe  and  track  targets  in  the 
surrounding  airspace. 

The  main  difficulty  that  arises  from  using  imaging  sensors  is  the  fact  that 
the  resulting  data  will  be  only  a  two-dimensional  representation  of  the  space 
around  the  aircraft,  and  as  such,  will  lack  any  range  data  for  the  targets  which 
must  be  tracked.  Without  range  data,  the  only  way  to  detect  an  impending 
collision  is  to  estimate  the  line-of-sight  (LOS)  velocity  of  the  tracked  targets 
and  look  for  any  target  with  a  nearly  zero  LOS  velocity. 

Section  2  outlines  the  method  of  extracting  measurements  from  the  image 
data  provided  by  the  earner a(s).  Section  3  outlines  the  motion  model  and 
tracking  method  used  to  maintain  the  track  on  the  target,  as  well  as  outlining 
the  method  of  initializing  the  track.  This  section  also  looks  at  the  consis¬ 
tency  of  the  tracker,  important  for  evaluating  the  “credibility”  of  a  collision 
warning.  Section  4  discusses  the  method  of  declaring  a  warning  of  possi¬ 
ble  collision.  Finally,  Section  5  discusses  why  comparison  of  the  extracted 
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measurements  to  GPS  data  (as  well  as  possible  hand-off  of  tracks  between 
multiple  cameras)  is  hampered  by  significant  (and  unknown)  lens  distortion. 

Additionally,  information  on  previous  measurement  extraction  and  target 
tracking  methods,  as  well  as  measurement  conversion  formulas,  can  be  found 
in  the  appendix. 


2  Measurement  Extractor 

In  order  to  track  targets  within  the  field  of  view  from  each  frame  of  a  camera, 
the  measurements  for  each  detected  target  must  be  extracted.  For  the  pur¬ 
poses  of  tracking  with  a  single  camera,  the  measurements  can  remain  pixel 
measurements  and  the  tracking  may  be  performed  in  the  pixel  space.  Track¬ 
ing  with  multiple  cameras,  however,  would  best  be  accomplished  by  tracking 
using  azimuth  and  elevation  measurements.  In  the  absence  of  any  distortion 
from  the  camera  lens,  the  pixel  measurements  could  be  easily  converted  to 
azimuth  and  elevation  measurements  (see  Appendix  A).  Distortion  from  the 
camera  lens,  however,  will  make  the  conversions  much  more  difficult  (see 
Section  5). 


2,1  Extraction  Algorithm 

Previous  methods  of  measurement  extraction  (outlined  in  Appendix  F.l) 
either  resulted  in  a  large  number  of  false  measurements  at  the  top  of  the 
frame  or  a  number  of  missing  measurements  from  true  targets  which  are  in  the 
frame  (although  they  are  unintentional  targets  which  appear  at  presumably 
much  greater  distances  than  the  test  target).  In  order  to  address  this,  another 
method  of  measurement  extraction  is  proposed. 

The  neighborhood  average  for  pixel  z,  denoted  as  7^,  will  be  calculated  as 


Ii  = 


LiL, 


E 

2  weWi 


(1) 


where  is  the  intensity  of  pixel  w,  L1L2  is  the  size  of  the  window,  and  Wi 
is  the  set  of  pixels  of  the  window  centered  at  pixel  i. 

In  addition  to  calculating  the  neighborhood  average  in  a  window,  the 
standard  deviation  of  the  intensities  in  the  window  is  also  calculated.  The 
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(2) 


standard  deviation  in  the  window  corresponding  to  the  -jth  pixel  is 


w€Wi 


1/2 


where  is  the  intensity  of  pixel  w,  L1L2  is  the  size  of  the  window,  is  the 
set  of  pixels  in  the  sliding  window  centered  at  pixel  i,  and  lyj  is  given  by  (1). 
A  detection  is  now  declared  at  pixel  i  if 


\Ii  -  Ii\  >  otOi 


(3) 


where  a  can  be  adjusted  to  tune  the  measurement  extractor. 

As  a  final  step  in  extracting  the  measurements,  the  centroid  of  each  group 
G  of  detected  pixels  is  calculated.  Detected  pixels  are  grouped  together  if 
they  share  a  side  or  corner  (i.e.,  they  are  connected  in  a  horizontal,  vertical 
or  diagonal  direction,  an  “8-connected  neighborhood”).  The  centroid  for  a 
group  of  pixels  is  calculated  within  a  rectangular  window  Wq  which  includes 
a  one-pixel  border  around  the  group  G.  The  centroid,  for  one  dimension,  is 

UG)  =  (4) 

where  Xi  is  the  a:-coordinate  of  pixel  i,  li  is  the  intensity  of  pixel  i,  and  the 
group  G  of  pixels  i  are  those  which  have  values  from  (3)  which  exceed  the 
threshold  and  have  been  grouped  together.  A  similar  expression  is  used  to 
calculate  the  ^-coordinate  of  the  centroid  for  group  G. 

Figure  1  shows  the  centroid  measurements  using  (3),  a  =  5  and  Li  =  L2  = 
101.  Preliminary  testing  seems  to  favor  a  larger  window  for  this  particular 
measurement  extractor.  The  use  of  such  a  large  window,  however,  causes 
long  computation  times  and  a  window  of  length  51  was  used  in  later  runs. 

Additionally,  extracted  measurements  which  are  found  below  the  horizon 
can  safely  be  ignored  since  there  is  no  danger  of  collision  with  a  target  which 
is  found  to  be  below  the  horizon.  This  can  help  eliminate  some  false  mea^ 
surements  arising  from  bright  spots  along  the  ground  in  some  of  the  video 
data  utilized  here. 


2.2  Second  Stage  Measurement  Extraction 

In  addition  to  utilizing  the  method  of  measurement  extraction  outlined  above, 
it  is  possible,  in  some  circumstances,  to  further  improve  the  measurement  ex¬ 
traction  by  repeating  a  similar  stage  of  measurement  extraction.  Since  the 
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Figure  1:  Measurements  (in  pixel  coordinates)  obtained  from  the  extractor  with  the 
5<7  rule  (ultralightl^rchive.vl,  scene  3,  BAE  camera) 


idea  behind  this  measurement  extraction  method  is  to  search  for  pixel  in¬ 
tensities  which  stand  out  from  the  background,  the  target  pixels  should  not 
be  included  amongst  the  background  pixels  during  detection.  The  problem, 
of  course,  is  that  detection  must  first  be  carried  out  in  order  to  know  which 
pixels  should  not  be  included  within  the  background.  In  an  attempt  to  mit¬ 
igate  this  “catch-22”  style  process,  a  two  stage  measurement  extraction  may 
provide  some  benefit. 

After  the  first  round  of  detection  is  carried  out  in  the  manner  outlined 
above,  the  detected  pixels  are  eliminated  from  the  calculation  of  the  (1)  and 
(2).  A  second  stage  of  detection  is  carried  out  in  exactly  the  same  manner, 
except  using  the  newly  calculated  U  and  Ui. 
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2»3  Variance  of  Extracted  Measurements 

2.3.1  Theoretical  Meeisurement  Noise  Variance  Calculation 

Rather  than  simply  make  an  assumption  about  the  measurement  noise  vari¬ 
ance,  it  is  possible  to  evaluate  the  measurement  noise  variance  during  mea¬ 
surement  extraction.  The  measurement  noise  variance  in  the  case  of  an  image 
centroid  (equation  (4)  above)  has  been  derived  previously  [1].  The  measure¬ 
ment  from  the  extractor  (equation  (4))  can  be  viewed  as 

Xc  =  Xc-\-  Wc  (5) 

where  Xc  is  the  true  image  centroid  in  the  ^^-coordinate  and  Wc  is  the  (zero 
mean)  centroid  measurement  noise.  Furthermore,  the  measured  pixel  inten¬ 
sity  is 

Ii  =  Si+  Tli  (6) 

where  Si  is  the  true  intensity  of  pixel  i  and  is  the  noise  intensity  in  pixel  i. 
The  variance  of  the  noise  in  the  centroid  measurement  can  be  evaluated 
in  a  window  of  the  image  as  [1] 

2  _ 

where  m  is  the  number  of  pixels  in  the  window,  r  is  the  SNR  in  the  window, 
li  is  the  intensity  of  pixel  i,  Wq  is  the  rectangular  window  around  the  group 
G  of  pixels  as  in  (4)  (i.e.,  a  single  target),  and  Xc  and  Xi  are  measured  using 
a  coordinate  system  centered  on  the  window.  The  value  of  the  SNR  in  the 
window,  r,  is 

r  =  (8) 

^/ma 

where  a  is  the  standard  deviation  of  the  image  noise. 

Since  the  value  Si  is  not  available  directly,  the  following  approximations 
can  be  used: 

E  ~  T.  (9) 

ieWo  ieWc 

E  ~  E 

i€WG  ieWa 

All  of  the  above  equations  assume  that  the  noise  intensity  is  zero  mean,  so 
the  estimated  noise  mean  must  first  be  subtracted  from  the  measured  pixel 
intensities  before  calculating  the  centroid  measurement  variance. 

5 


45 


2.3.2  Empirical  Measurement  Noise  Variance  Estimation 

Rather  than  relying  on  theoretical  calculations  for  the  measurement  variance 
of  the  centroid  of  an  IR  target,  there  is  another  possible  method  of  variance 
estimation  that  can  be  used.  Previously  used  to  estimate  the  unknown  mea¬ 
surement  noise  variance  of  radar  systems  (see  [3]),  this  method  uses  multiple 
Interacting  Multiple  Model  (IMM)  estimators  in  order  to  estimate  the  un¬ 
known  variance.  The  use  of  an  IMM  estimator  is  not  specifically  required 
by  this  method,  however,  IMM  estimators  often  lead  to  very  good  tracking 
performance  for  maneuvering  targets. 

In  order  to  estimate  the  unknown  measurement  variance,  multiple  IMM 
estimators  (three  in  this  case)  must  be  run  simultaneously,  each  with  its 
own  assumption  for  the  measurement  noise  variance.  Each  estimator  should 
also  be  identical  except  for  the  assumed  measurement  noise  variances.  For 
the  scope  of  this  work,  the  three  estimators  were  set  up  with  one  using  an 
appropriate  initial  variance,  and  the  other  two  using  scaled  versions  of  that 
variance  (50%  and  150%  of  the  assumed  variance). 

In  order  to  determine  the  most  likely  measurement  noise  variance  as¬ 
sumption,  the  likelihood  of  each  estimator  must  be  determined.  Each  IMM 
estimator  calculates  the  probability  of  the  system  being  in  each  mode  as¬ 
sumed  by  the  estimator  (details  on  the  IMM  estimator  can  be  found  in  [2]). 
The  mode  probabilities  can  then  be  used  to  evaluate  a  likelihood  function 
for  each  IMM  estimator. 

The  probability  that,  according  to  IMM  number  m,  the  system  is  in  mode 
i  at  time  k,  given  the  data  at  time  k  —  1,  with  =  1, . . . ,  r  is 

-  1)  =  -  1)  m  =  (11) 

i=i 

where  /i^(^  —  1)  is  the  probability  of  being  in  mode  j  at  /c  —  1,  and  Pji  is  the 
probability  of  transition  from  mode  (motion  model)  j  to  mode  i. 

The  likelihood  function  for  IMM  estimator  m  at  time  k  can  then  be  writ¬ 
ten  in  terms  of  the  likelihood  functions  for  each  mode  of  this  IMM  estimator 
and  the  above  probabilities,  namely  [2], 

A™[^(/£)1  =  j2Wk);zUk\k  -  \),Slik)]nl(k\k  -  1)  (12) 

i=l 

wheie  Af[z{k);zl^{k\k  —  l),Sl^{k)]  is  a  Gaussian  pdf  that  represents  the  likeli¬ 
hood  function  of  mode  i  of  IMM  m  at  time  /c,  is  the  predicted  measurement 
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for  filter  %  and  is  the  innovation  covariance  of  filter  %,  In  other  words, 
the  likelihood  function  of  the  IMM  estimator  is  the  sum  of  the  likelihood 
functions  of  each  filter  multiplied  by  the  one-step  predicted  probability  of 
being  in  that  mode. 

As  more  data  are  obtained,  the  likelihood  function  for  each  IMM  esti¬ 
mator  can  be  updated  in  the  form  of  a  cumulative  likelihood  function,  given 
as 

=  ri  ^m[z{k)]  (13) 

k=ki 

where  ki  is  the  starting  time  and  k2  is  the  ending  time.  The  probability  of  the 
mth  IMM  estimator  (measurement  model  M^)  having  the  correct  variance 
assumptions  given  the  data  up  to  and  including  k2  can  then  be  obtained  as 

A  A  ^2 

(14) 


which  assumes  a  uniform  prior. 

The  probability  from  each  estimator  can  then  be  used  as  a  weighting  to 
arrive  at  the  estimated  measurement  noise  variance,  i.e., 

m=l 

where  n  is  the  number  of  IMM  estimators  used  {n  =  3,  in  our  case).  This 
variance  can  then  be  used  by  the  ‘‘center”  estimator,  while  the  others  use 
scaled  versions  of  the  newly  calculated  value.  This  process  continues  as  the 
estimators  track  each  target  and  result  in  a  continuously  updated  measure¬ 
ment  noise  variance. 

A  method  of  measurement  noise  variance  estimation  such  as  this  is  nec¬ 
essary  for  tracking  in  this  passive  collision  warning  system  due  to  the  non¬ 
stationary  nature  of  the  measurement  noise.  For  example,  as  targets  move 
closer  or  farther  from  the  camera,  the  measurement  noise  will  change  as  the 
target’s  energy  is  spread  over  a  different  number  of  pixels.  Additionally,  as 
the  target  moves  throughout  the  scene,  the  environment  will  have  differing 
effects  on  the  target  detection  and  measurement  noise. 
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3  Target  Tracker 

Once  the  measurements  corresponding  to  each  detection  centroid  in  a  frame 
of  data  are  obtained,  the  tracker  can  use  these  measurements  to  start  tracks 
and  estimate  target  trajectories.  The  target  state  consists  of  position,  veloc¬ 
ity  and  acceleration  in  the  focal  plane  (FP).  Currently,  a  nearest  neighbor 
approach  is  utilized  to  assign  measurements  to  tracks.  Each  currently  run¬ 
ning  track  is  assigned  the  measurement  which  is  closest  to  the  predicted 
measurement  location  (provided  it  falls  within  a  validation  region  centered 
at  the  predicted  measurement  [1]).  Currently  a  5a  validation  gate  is  applied 
to  the  measurements.  The  number  of  measurements  in  a  validation  gate  is 
rarely  greater  than  one,  so  using  a  measurement  assignment  method  more 
complex  than  the  nearest  neighbor  approach  is  not  necessary.  Once  measure¬ 
ments  are  assigned  to  the  currently  running  tracks,  each  track  is  updated  via 
an  IMM  estimator  [2]  utilizing  two  Kalman  filters  which  assume  appropriate 
motion  models. 

3.1  Target  Motion  Models 

3.1.1  Autocorr elated  Process  Noise  Model 

Standard  nearly  constant  velocity  (NCV)  and  nearly  constant  acceleration 
(NCA)  motion  models  assume  that  the  process  noise  (acceleration  in  the 
NCV  model  and  acceleration  increment  in  the  NCA  model)  can  be  treated 
as  white.  In  the  case  of  video  data,  the  sampling  time  is  1/30  s,  and  the  white 
process  noise  assumption  becomes  unrealistic.  The  accelerations  the  target 
undergoes  clearly  must  be  correlated  between  samples  when  the  sampling 
time  is  very  small.  For  this  reason  an  autocorrelated  process  noise  model  will 
provide  higher  quality  target  tracking  than  NCV  or  NCA  motion  models. 

Specifically,  the  autocorrelated  process  noise  model  used  here  models  the 
target  accelerations  as  an  exponentially  autocorrelated  noise  with  zero  mean 
[2],  The  autocorrelation  of  the  target  acceleration  a{t)  is  therefore 

E  [a(t)a(i -h  r)]  =  a  >  0  (16) 

where  cr^  is  the  instantaneous  variance  of  the  acceleration  and  I /a  is  the 
time  constant  of  the  target  acceleration  autocorrelation.  For  this  model,  the 
state  equation  will  be 

x{k  +  1)  =  Fx{k)  i- v{k)  (17) 
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where 


F 


■  1  T 
0  1 
0  0 
0  0 
0  0 

.  0  0 

■  1  T 
0  1 
0  0 
0  0 
0  0 
0  0 


(aT-l-he-"^)/a2  0  0 

(1— e“"^)/a  0  0 

0  0 

0  1  T 

0  0  1 

0  0  0 

0  0  0  0  ' 

TOO  0 
1-aT  0  0  0 

0  IT  0 
0  0  1  T 

0  0  0  l-aT 


0 

0 

0 


(aT-1  4-e-“^)/a2 
(1  —  e~^'^)/a 


(18) 


Additionally,  the  process  noise  v{k)  will  have  a  covariance  matrix  of 

■  TV20  TVs  TV6  0  0  0  * 

TVs  TVs  T2/2  0  0  0 

^  ^  fl9) 

TV20  TVs  TVs  ^  ^ 

TVs  TVs  TV2 

TVs  TV2  T 

A  value  of  a  =  2s“^  was  deemed  appropriate  for  both  motion  models  of 
the  IMM  estimator  as  it  results  in  a  decorrelation  time  of  roughly  1  second. 


Q  =  2a(j; 


TVs  TV2  T 

0  0  0 

0  0  0 

0  0  0 


3.2  Choice  of  the  Process  Noise  Level 

The  standard  deviation  of  the  process  noise,  cr^,  should  be  chosen  to  cover 
the  expected  accelerations  of  the  targets.  Early  tests  were  run  using  three 
scenes  of  an  ultralight  aircraft  passing  over  the  camera.  The  supplied  data  for 
the  BAE  camera  was  examined  to  determine  the  test  target’s  accelerations 
both  near  to  and  far  from  the  camera.  The  position  of  the  test  target  was 
found  over  two  one-second  intervals  in  order  to  estimate  the  velocity  and 
acceleration  of  the  target,  the  difference  in  position  providing  velocity  and 
the  difference  of  two  velocity  estimates  providing  the  acceleration  estimate. 
The  estimated  accelerations  are  shown  in  Table  1.  Scene  4  was  estimated  to 
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Ultralight  1  .Archive,  vl 
BAE  Camera 

Near  Camera 

Far  Prom  Camera 

Scene  2 

|a|  = 

(U 

|a|  = 

00  00 

0  0 

Scene  3 

|a|  =  1 

54.39 

11.4 

Scene  4 

|a|  =  | 

32.3  ) 

\ 

1 

|a|=| 

p  0 
to  ^ 

Table  1:  Estimated  acceleration  of  the  test  target  from  the  BAE  Camera  in  various 
scenes 


have  a  zero  acceleration  in  the  y  direction  for  the  “near  camera”  section  since 
the  velocity  estimates  were  found  to  be  equal  in  that  portion  of  the  video. 

Based  on  the  estimated  accelerations  of  the  test  target,  a  value  of  =  10 
pixels/s^  was  used  in  some  simulations  on  Scene  3  using  a  single  Kalman  filter 
with  an  autocorrelated  process  noise  model  with  a  =  0.5 s“^.  The  difference 
in  acceleration  values  between  the  near  and  far  targets  as  well  as  between 
different  scenes,  however,  suggested  that  a  multiple  model  approach  would 
be  needed  eventually  for  better  tracking  across  different  scenarios. 

In  the  case  of  the  IMM  estimator  w^hich  utilized  two  autocorrelated  pro¬ 
cess  noise  models,  parameters  of  ai  =^a2  =  2s“^  and  =  25  and  <7^2  ^ 

were  used.  It  should  be  noted  that,  at  this  stage,  the  L3  camera  was  used 
exclusively;  and  the  lower  resolution  of  the  L3  camera,  as  well  as  the  different 
a  values,  leads  to  the  use  of  different  process  noise  levels.  In  general,  for  an 
autocorrelated  process  noise  motion  model,  the  choice  of  am  is  related  to  the 
process  noise,  g,  of  a  white  noise  acceleration  (WNA)  model  by  [2] 

=  Q  (20) 

a 


3.3  Track  Initialization 

In  order  to  initialize  tracks,  a  validation  gate  (in  pixel  space),  based  on  the 
assumed  maximum  velocity  of  a  target  over  one  sampling  period,  is  set  up  for 
each  measurement  location  for  the  potential  tracks.  Originally,  the  assumed 
maximum  velocity  of  100  pixels/s  in  the  2:-direction  and  50  pixels/s  in  the 
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^-direction  was  used  when  the  target  was  entering  the  camera  frame  at  a 
relatively  close  distance  (and  was  therefore  much  larger  in  number  of  pixels). 
The  maximum  velocity  assumption  is  important  only  for  track  initialization, 
however;  and  for  a  realistic  scenario,  a  target  of  interest  will  be  initialized  at 
a  much  greater  distance,  and  will  therefore  have  a  much  smaller  maximum 
velocity  in  pixel  space.  For  this  reason,  the  track  formation  using  maximum 
velocity  assumptions  of  10  or  20  pixels/s  (in  both  x  and  y  directions)  are 
likely  more  reasonable. 

Additionally,  instead  of  insisting  that  two  consecutive  measurements  be 
found  (as  described  in  section  Appendix  H.l),  the  track  initialization  will 
instead  look  for  three  valid  measurements  within  five  sampling  intervals. 
This  amounts  to  an  M/A'  logic  [1]  with  M  =  3,  A  =  5  and  will  allow  for 
intermittent  detection  of  a  target  which  is  a  large  distance  from  the  camera. 
In  this  case,  the  validation  gate  must  be  increased  when  there  is  a  missed 
detection.  The  increased  gate  size  should  not  cause  difficulties  for  the  tracker 
as  the  initial  gate  is  small  due  to  the  assumption  of  the  target  being  at  far 
distances  (when  track  initialization  should  be  taking  place)  and  therefore 
appearing  as  a  small  low- maneuvering  target.  For  the  case  of  “three  out  of 
five”  track  initialization,  the  validation  gate  will  cover,  at  worst,  three  times 
the  maximum  velocity  over  a  single  sampling  interval. 

Following  track  maintenance  (see  section  3.4)  all  the  remaining  measure¬ 
ments  in  the  frame  are  used  to  initialize  new  possible  tracks.  All  unassigned 
measurements  from  a  frame  are  saved  as  starting  points  for  new  potential 
tracks  at  the  next  sampling  time. 

3.3.1  Comparison  of  Track  Initialization  Methods 

Table  2  compares  the  number  of  tracks  formed  and  time  of  track  confirma¬ 
tion  for  each  of  the  track  initialization  procedures  (2/2  and  3/5  logic).  The 
track  initialization  was  run  using  the  data  from  the  BAE  camera,  movie  file 
“Ultralight  1  Archive. vl,”  scene  3,  the  measurement  extractor  with  window 
length  51  and  a  5(7  threshold,  and  the  older  WPA  motion  model  (see  section 
Appendix  G.l). 

In  each  case,  the  test  target  track  was  always  formed  and  the  extra  tracks 
were  the  result  of  measurements  that  are  the  result  of  intermittent  bright 
spots  along  the  tree  line  or  clouds,  or  the  result  of  a  flock  of  birds  passing 
through  the  camera  frame  (e.g.,  the  upper  part  of  Figure  1). 
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\Xmax,Vmax\  (in  plxels/s) 

Number  of  Tracks 

[100,50] 

[20, 20] 

[10, 10] 

2/2  Initiator 

491 

140 

137 

3/5  Initiator 

320 

81 

77 

Test  Track  Confirmation  Fi*ame 

2/2  Initiator 

1888 

1888 

1888 

3/5  Initiator 

1887 

1887 

1887 

Table  2:  Comparison  of  number  of  tracks  formed  for  various  track  initialization  pro¬ 
cedures 


3.4  Track  Mainteiiaiice 

Once  potential  tracks  have  been  confirmed,  they  are  treated  as  if  they  are 
a  true  track.  A  validation  gate  is  set  up  at  each  currently  running  track’s 
predicted  measurement  location.  The  closest  measurement  to  the  predicted 
measurement  location  that  falls  within  the  validation  gate  is  assigned  to 
that  track.  The  measurement  is  then  used  in  an  IMM  estimator,  which 
passes  the  measurement  to  two  standard  Kalman  filters  with  autocorrelated 
process  noise  models  as  discussed  in  3.1.  After  the  currently  running  tracks 
are  assigned  measurements  and  propagated,  the  remaining  measurements 
of  the  frame  are  used  in  initialization  of  new  tracks  or  saved  as  potential 
starting  points  of  new  tracks  for  the  next  frame.  If  a  track  does  not  have 
a  measurement  assigned  to  it,  the  track  will  be  propagated  “open- loop.”  If 
four  consecutive  missed  detections  occur,  the  track  will  be  terminated. 

3.5  Distance  at  Track  Confirmation 

In  order  to  determine  the  distance  at  which  the  test  flight  target  is  acquired, 
the  frame  number  at  track  confirmation  is  recorded.  The  timestamp  of  that 
frame  can  then  be  compared  to  available  GPS  data  to  determine  the  location 
of  the  target  of  interest.  Use  of  the  equations  of  Appendix  C,  with  the  camera 
location  as  the  origin,  will  result  in  the  ENU  vector  of  the  target  at  the  time 
of  track  confirmation.  Simply  taking  the  magnitude  of  this  vector  results  in 
the  distance  from  the  camera  to  the  target. 

The  following  distances  of  track  confirmation  utilize  a  measurement  ex¬ 
tractor  with  Li  =  L2  =  51  and  ba  threshold,  and  a  single  filter  tracker  with 
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autocorr elated  motion  model  with  a  =  0.5  s“^  and  =  lOpixels/s^.  The 
camera  is  assumed  to  be  located  at  .39°32'31.82"  N  latitude,  75°  18' 14.5''  W 
longitude  and  0  elevation  (in  the  absence  of  altitude  information).  The  dis¬ 
tance  at  track  confirmation  using  the  BAE  camera  is  shown  in  table  3  (Note: 
the  video  is  played  backwards  to  simulate  a  target  approaching  the  camera 
rather  than  flying  away  from  the  camera). 


BAE  Camera 

Confirmation  Frame 

GPS  Coordinates  |  Distance 

Scene  2 

1814 

39°  33'  2.9375"  N 
75°  18'  19.9005"  W 
305  ft 

973  m 

Scene  3 

1887 

39°  33'  1.8832"  N 
75°  18'  20.1322"  W 
291  ft 

941  m 

Scene  4 

2657 

39°  33'  25.7443"  N 
75°  18'  31.3793"  W 
669  ft 

1,723  m 

Table  3:  Distance  to  target  of  interest  at  track  confirmation  for  various  scenes 


3.6  Target  Tracker  Consistency 

Since  the  estimated  state  from  the  target  tracker  will  be  utilized  to  test 
whether  or  not  a  target  is  on  an  impending  collision  course,  it  is  important 
that  the  target  tracker  be  consistent^  [2].  Since  the  consistency  must  be  tested 
from  a  single  run  (since  the  tracker  is  operating  on  real  data)  the  consistency 
of  the  target  tracker  must  be  tested  with  the  normalized  innovation  squared 
(NIS).  The  normalized  innovation  squared  is 

ei,(k)  =  i/(kys(k)~^i/(k)  (21) 

where  i/(k)  is  the  innovation  at  time  k  and  S(k)  is  the  innovation  covariance 
at  time  k.  If  the  filter  is  consistent,  the  NIS  will  have  a  chi-square  distribution 
with  degrees  of  freedom,  where  is  the  dimension  of  the  measurement 

^Consistency  of  a  state  estimator  is  the  requirement  that  its  calculated  state  estimate 
variances  are  statistically  compatible  with  the  actual  errors. 
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(in  our  case,  n, 
NIS 


2).  In  order  to  test  from  a  single  run,  the  time-averaged 


=  (22) 

^  fe=l 

may  be  used,  where  Kit,  follows  a  chi-square  distribution  with  Kuz  degrees 
of  freedom  [2],  In  the  case  of  a  time- aver  aged  NIS  over  100  samples,  the  95% 
confidence  region  will  be  between  1.63  and  2.41. 

In  the  case  of  an  IMM  estimator,  there  are  two  (or  more,  in  general) 
differing  motion  models,  which  each  result  in  different  values  for  (21).  In 
this  case,  the  NIS  will  be  approximated  as  a  combination  of  each  mode’s 
NIS,  et,i{k),  where 


(23) 

and  will  be  expressed  as 

(24) 

i=l 


where  r  is  the  number  of  modes  in  the  IMM  estimator  and  is  the  probability 
that  the  system  is  in  mode  i.  The  time-averaged  NIS  can  now  be  calculated 
as  before,  using  (24)  in  place  of  (21). 

3.6.1  Fixed  Centroid  Measurement  Noise 

Figure  2  shows  the  sliding  window  time-averaged  NIS  for  a  single  model 
filter  using  an  autocorrelated  process  noise  model  with  a  =  0.5  s“^  CTm  ~ 
lOpixels/s^,  and  <Jc  =  0.5  pixel.  Figure  3  shows  the  time-averaged  NIS  with 
the  same  parameters,  except  that  measurement  noise  standard  deviation  (Jc  = 
0.1  pixel.  Both  filters  were  run  on  the  data  from  Scene  3  from  the  BAE 
camera.  Both  figures  show  the  time-averaged  NIS  over  a  sliding  window  of 
100  samples. 

When  the  measurement  noise  standard  deviation  is  taken  as  0.5  pixels, 
the  average  NIS  is  clearly  far  from  the  values  expected  for  a  100  sample 
average  NIS.  Since  the  NIS  is  much  lower  than  expected,  the  innovation  co- 
variance  must  be  too  large,  suggesting  that  the  measurement  noise  standard 
deviation  was  too  large.  When  the  measurement  noise  standard  deviation  is 
lowered  to  0.1  pixels,  the  average  NIS  takes  on  values  much  closer  to  what  is 
expected.  Furthermore,  the  movement  of  the  NIS  may  also  suggest  that  the 
measurement  noise  standard  deviation  may  vary  throughout  the  scene.  If 
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100  Sample  Sliding  Average  NIS 
a^=10,  0^=0  5 


1.8h 

1.6- 

1.4- 

1.2- 


Figure  2:  100  Sample  sliding  window  time-averaged  NIS  with  a^n 
(Jc  =  0.5  pixel 


900 

=  10  pixels/s^, 


the  measurement  noise  does  indeed  change  depending  on  the  scenario,  then 
a  tracker  which  can  adapt  to  these  changes  would  be  preferable. 

Figures  4  and  5  show  the  sliding  window  time-average  NIS  for  a  single 
model  filter  with  an  autocorrelated  process  noise  model  with  a  =  0.5  s“^, 
=  2.5  pixels/s^,  and  <7^  calculated  using  (7).  Figure  4  uses  data  from 
camera  B,  scene  “Loop  1”  of  the  “Piperl.vP  movie  file.  Figure  5  uses  the 
data  from  camera  A  of  the  same  scene. 

3.6.2  Dynamically  Determined  Centroid  Measurement  Noise 

By  using  the  theoretical  calculations  for  the  variance  of  the  centroid  mea^ 
surement  noise  (equation  (7)),  the  tracker  can  utilize  a  measurement  noise 
standard  deviation  which  is  free  to  change  throughout  the  scenario.  Addi¬ 
tionally,  since  the  tracker  is  evaluating  the  measurement  noise  from  the  data, 
each  camera  will  be  able  adjust  to  the  particular  view  it  has  of  the  target. 
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100  Sample  Sliding  Average  NIS 
a^=10.  0^=01 


Figure  3:  100  Sample  sliding  window  time-averaged  NIS  with  a^n  =  lOpixels/s^, 
cTc  =  0.1  pixel 


Different  cameras  may  detect  a  different  number  of  pixels  for  each  target,  for 
example,  so  the  centroid  measurement  noise  shouldn’t  be  assumed  to  be  the 
same  across  cameras. 


4  Threshold  Test  for  Collision  Course 

Lacking  any  range  information,  the  only  way  to  test  whether  the  target  is 
on  a  collision  course  is  to  examine  the  estimated  angular  (LOS)  velocity  of 
the  target.  For  the  case  of  tracking  a  target  in  pixel  space,  the  velocity 
estimates  in  the  two  pixel  dimensions  (x  and  y)  can  be  used.  A  target 
which  has  an  estimated  angular  velocity  near  zero  can  be  assumed  to  be  on 
a  collision  course.  As  long  as  either  x  or  ^  is  estimated  to  be  far  enough  from 
zero  (greater  than  3a  away),  no  collision  will  be  declared.  In  order  to  set  a 
threshold  for  declaration  of  an  impending  collision,  the  filter-calculated  track 
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Figure  4:  100  Sample  sliding  window  time-averaged  NIS  with  cr^n  =  2.5  pixels/s^, 
calculated,  camera  B 


covariance  will  be  used  to  compare  the  estimated  velocity  to  its  standard 
deviation. 

Figures  6  and  7  show  example  plots  of  ^  and  respectively,  with  the 
dashed  lines  denoting  the  3(7  boundaries.  The  data  in  this  case  is  from  a 
single  model  tracker  using  autocorrelated  process  noise  with  a  measurement 
noise  standard  deviation,  (Jc,  assumed  to  be  0.1,  which  results  in  the  more 
consistent  filter  as  seen  in  figure  3.  When  both  estimated  velocities  are  found 
to  be  within  3a  of  zero,  a  collision  could  be  declared.  As  the  track  continues, 
however,  it  is  clear  that  no  collision  will  occur  as  at  least  one  of  the  estimated 
velocities  is  more  than  3a  away  from  zero.  In  this  case,  the  estimate  of  x 
is  almost  never  within  3a  of  zero,  so  there  will  be  no  collision  warning  from 
this  target. 
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Figure  5:  100  Sample  sliding  window  time-averaged  NI5  with  cr^  =  2.5  pixels/s^, 
(Jc  =  calculated,  camera  A 


5  Comparison  Between  Calculated  Track  and 
GPS  in  LOS  Space 

In  order  to  compare  the  extracted  measurements  and  the  calculated  track  to 
the  available  GPS  data  of  the  target,  the  measurements  must  be  converted 
from  pixel  space  to  LOS  space.  Appendix  A-Appendix  D  outline  the  neces¬ 
sary  calculations  and  conversions  for  comparing  the  calculated  track  to  GPS. 
These  are  done  assuming  a  distortionless  lens  where  a  pixel  location  can  be 
converted  directly  to  a  LOS  angle. 

Additionally,  the  camera  boresight  angles  and  the  roll  of  the  camera  can 
be  estimated  by  comparing  the  measurements  from  the  camera  to  the  avail¬ 
able  GPS  data.  Figure  8  shows  the  GPS  data  from  the  Piperl.vl  file  scene 
“Loop  1”  in  azimuth  and  elevation  angles  as  well  as  the  measurements  from 
camera  B  converted  to  azimuth  and  elevation  angles.  The  offset  of  the  two 
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i/<Ti 


Figure  6:  Collision  threshold  test  on  the  estimate  x  —  ratio  of  the  velocity  estimate 
to  its  standard  deviation  estimate:  "non-zero  velocity  estimate"  "no  collision" 
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Figure  7:  Collision  threshold  test  on  the  estimate  y  —  ratio  of  the  velocity  estimate 
to  its  standard  deviation  estimate 
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Figure  8:  Comparison  of  GPS  data  with  measurement  conversions  to  LOS,  Camera 
B,  Piperl,  Loop  1 


plots  represent  the  azimuth  and  elevation  angles  of  the  camera  boresight  as 
well  as  a  possible  non-zero  roll  angle  of  the  camera.  Figure  9  shows  simi¬ 
lar  data  for  the  same  scene  with  camera  A.  Both  figures  also  highlight  the 
portions  of  the  data  for  which  the  frames  of  the  two  cameras  are  overlapped. 

Unfortunately,  in  both  cameras,  there  is  a  “barrel”  distortion  clearly 
present  at  the  edges  of  the  frame,  which  causes  the  converted  camera  pixel 
measurements  to  match  poorly  to  the  LOS  based  on  GPS  data.  This  causes 
the  overlapped  portion  of  the  two  cameras  to  exhibit  a  small  amount  of  roll 
when  in  fact  camera  A  is  rolled  a  great  deal  with  respect  to  camera  B.  The  dis¬ 
tortion  at  the  edge  of  the  frames,  therefore,  will  hamper  relative  registration 
and  there  can  be  no  handoff  of  target  tracks  from  one  camera  to  the  other. 
Additionally,  comparison  to  GPS  data  is  not  possible,  due  to  the  unknown 
distortion  mapping  between  apparent  focal  plane  and  true  azimuth/elevation 
coordinates,  this  discrepancy  being  largest  near  the  image  boundaries. 
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Camera  A 


Figure  9:  Comparison  of  GPS  data  with  measurement  conversions  to  LOS,  Camera 
A,  Piper  1,  Loop  1 
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APPENDIX 


Appendix  A  Conversion  of  pixel  location  to 
azimuth  and  elevation 


We  wish  to  convert  measurements  in  pixel  space  to  measurements  in  angle 
space.  The  following  derivation  shows  the  conversion  of  pixel  [x,y)  to  az¬ 
imuth  and  elevation  angles  a  and  e,  respectively  (measured  about  the  frame 
boresight).  For  an  image  of  width  pixels  and  height  Ny  pixels,  the  carte¬ 
sian  coordinates  of  a  pixel  will  be  numbered  left  to  right  (x  =  1, . . . ,  N^)  and 
bottom  to  top  (?/  =  1, . . . ,  Ny),  More  traditional  pixel  location  representation 
(the  [1,1]  pixel  located  in  the  top  left  corner  of  the  image)  can  be  used  by 
simply  utilizing  the  transformation  Xi  =  x  and  yi  =  Ny  —y  1,  The  camera 
is  assumed  to  have  a  horizontal  aperture  angle  Ax  and  vertical  aperture  angle 
Ay,  focal  length  of  /,  and  square  pixels  of  width  p. 

The  distance  to  the  rightmost  edge  of  the  image  is  given  as 

(25) 

and  the  horizontal  distance  to  the  center  of  pixel  {x,y)  is 

(x-^-^)p  =  fta.ua  (26) 

Now  using  (25)  and  (26)  we  have 

a  =  tan->(tan(^)(^^-^--^))  (27) 


In  a  similar  fashion  the  distance  to  the  topmost  edge  of  the  image  is  given 


1  .  Ay 

-Nyp=  f  tan  ^ 

The  vertical  distance  to  the  center  of  pixel  (x,  y)  is 

_  i\  „  =  f 

2  2r  cosQ 

where  a  is  taken  from  (27).  Relating  (28)  and  (29)  we  now  have 


(28) 


(29) 


e  =  tan-i  ( tan  cos  (a)  ^ 


(30) 
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Now  that  we  have  the  measurement  in  terms  of  azimuth  and  elevation 
in  the  Focal  Plane  (FP)  frame,  we  wish  to  express  the  measurement  as  a 
line-of-sight  (LOS)  unit  vector  in  the  FP  coordinate  system.  The  LOS  unit 
vector  relative  to  the  FP-xyz  frame  is 


LOS  =  -f  lyLy  -r  UL,  (31) 

where  =  cos  e  sin  a,  Ly  =  sin  €  and  =  cos  €  cos  a. 

Appendix  B  Conversion  of  LOS  in  FP  coor¬ 
dinate  system  to  local  ENU  co¬ 
ordinate  system 

Next  we  must  convert  the  line  of  sight  (LOS)  vector  in  the  focal  plane  (FP) 
coordinates  to  east-north-up  (ENU)  coordinates  (centered  at  the  camera)  in 
order  to  compare  the  track  with  GPS  information.  We  first  assume  that  the 
frame  boresight  has  an  elevation  of  ep  (above  the  horizontal),  an  azimuth 
angle  of  a p  (in  the  horizontal  plane  clockwise  from  north),  and  a  roll  angle 
of  pp  (in  the  vertical  plane  clockwise  from  up).  In  order  to  transform  between 
FP  frame  coordinates  and  ENU  coordinates  we  first  assume  that  the  frame 
(lx/'/,  ly///,  Iz'//)  coincides  with  1e,  In  and  lu-  Using  three  rotations,  we  can 
transform  any  FP  frame  of  given  azimuth,  elevation  and  roll  to  local  ENU 
coordinates. 

First,  a  rotation  of  180  — ai?’  along  Ig///  yields  an  azimuth  pointing  of  ap. 


This  rotation  is  given  as 

lx/'  =  lx'/'  cos(180  —  ap)  ly///  cos{ap  —  90) 

=  lx"/(— cos  Q/p) -h  ly/// sin  ct/r  (32) 

ly/,  =  cos(90  +  180  —  a p)  ly///  cos(180  —  ap) 

=  lx///(— sin  o;^) -j- ly///(— cos  Qfjr)  (23) 

Iz"  =  Iz'"  (34) 

Next,  a  rotation  of  90  —  ep  along  lx//  tilts  the  frame  to  an  elevation  of  ep. 
This  rotation  is  given  as 

lx'  =  lx"  (35) 
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(36) 


ly/  =  ly//  COS (90  —  ep)  \^n  cos(eir) 

=  ly// sin  CiP  +  Iz/^  cosejr 
Ijj/  =  ly/t  cos(180  —  ep)  +  Ig//  cos(90  —  ep) 


=  ly/^(— cosep)  +  Iz// sinciP  (37) 

Finally,  a  rotation  of  pp  around  l^i  tilts  the  frame  to  the  proper  roll  angle. 
This  rotation  is  given  as 

lx  Ix^  cos  Pp  -h  ly/  sin  pp  (38) 

ly  =  lx/  cos(90  -f  Pp)  +  ly/  cos  pp 

=  —  Ix/sinpp  T  ly/cosp/p  (39) 

Iz  =  Iz/  (40) 


Combining  these  rotations  together  we  have  the  transformation  of  the 
measurements  of  azimuth  and  elevation  angles  within  the  FP  frame  to  a 
LOS  unit  vector  in  ENU  coordinates.  The  full  transformation  is  given  as 


where 


LOS  =  \l^  Ly  L/ 


—  Lx  Ly  Lz  Ts 


lx' 

]T3 

ly' 

lz' 

lx// 

]TsT2 

ly// 

.  lz'  . 

—  ^Lx  Ly  L2  j  T3T2T1 
=  ^  Lx  Ly  Lz  T2T2T1 

Le  Ln  Lu  \  =  ^  Lx  Ly  L2  J  T3T2T1 


ly/// 

lz/// 

1e 

In 

lu 


cospp  sinpp  0 
—  sinpp  cospp  0 
0  0  1 


(41) 

(42) 

(43) 

(44) 

(45) 

(46) 

(47) 
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1 

0 

0 

0 

smep 

cos  ep 

(48) 

0 

—  cos  ep 

sin  ep 

- 

cos  ap 

sin  ap 

0  ■ 

- 

sinap  - 

-  cos  ap 

0 

(49) 

0 

0 

1 

Appendix  C  Conversion  of  GPS  coordinates 
to  local  ENU 

In  order  to  evaluate  the  accuracy  of  tracking,  GPS  measurements  from  test 
targets  will  be  provided  along  with  the  video  data  from  the  cameras  used. 
In  order  to  properly  compare  this  data,  the  GPS  coordinates  must  be  trans¬ 
formed  into  the  same  ENU  coordinate  system  used  for  the  measurement 
conversion  (ENU  centered  at  camera).  In  order  to  convert  geodetic  (GPS) 
measurements  to  ENU  coordinates,  we  must  first  convert  the  measurments 
(as  well  as  the  GPS  coordinates  of  the  origin)  to  earth-centered  earth-fixed 
(ECEF)  coordinates. 

For  GPS  coordinates  of  height  h,  latitude  A,  and  longitude  (/>,  the  ECEF 
coordinates  (x,y^z)  are  given  by 

X  =  (h  +  N)  cos  A  cos  (!>  (50) 

y  =  (/i  +  AT)  cos  A  sin  (51) 

^  (52) 

where  N  =  >  ^  e  =  8.1819191  x  10~^  (eccentricity  term)  and  a  = 

y/  l-e^sin^  A 

6378137  (semimajor  axis  of  earth,  in  meters). 

The  conversion  of  ECEF  coordinates  {x,y,z)  to  local  ENU  coordinates 
(e,n,w)  is  given  by 


e 

—  sm(j>  cos(t>  0 

’  X  —  Xq 

n 

= 

—  cos  <;/!)  sin  A  —  sin  A  sin  ^  cos  A 

y-va 

(53) 

u 

cos  A  cos  (f>  cos  A  sin  (j)  sin  A 

_  Z-  Zq  _ 

where  (xo,i/o,^o)  is  the  origin  of  the  local  ENU  coordinate  system  in  ECEF 
coordinates,  given  by  (50)-(52)  with  A  =  /iq,  A  =  Aq,  0  =  GPS 

coordinates  of  the  camera. 
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Appendix  D  Comparison  of  track  from  cam¬ 
era  to  GPS  information 

The  overall  procedure  for  comparing  the  track  generated  from  the  camera 
measurements  to  the  GPS  information  follows  the  flow  chart  provided  here. 


Also,  note  that  once  the  GPS  coordinates  of  the  target  are  converted  to 
ENU  coordinates  (with  the  origin  centered  at  the  camera)  the  distance  from 
camera  to  target  is  simply  the  magnitude  of  the  ENU  vector  of  the  target. 


Appendix  E  Positioning  of  two  fixed  cameras 
for  PCAS 


In  order  to  calculate  the  azimuth  overlap  in  field  of  view  (FOV)  of  two  cam¬ 
eras  (assumed  to  have  the  same  boresight  elevation  angles),  their  pointing 
angles  (pi,  P2  in  Figure  10,  assumed  given  in  degrees)  must  be  known.  The 
overlap  e  (in  degrees)  of  the  two  cameras  is 


=  Pi  +  Y- 


a2 

P2-y 


(54) 


where  cti  and  a2  are  the  horizontal  FOV  of  each  camera,  and  pi,  p2  are  their 
boresight  angles  (measured  clockwise  with  respect  to  the  perpendicular  on 
the  base  L). 

For  two  cameras  separated  by  a  distance  L,  the  distance  d  in  Figure  10 
(the  distance  from  the  cameras  to  the  intersection  of  their  fields  of  view)  is, 
for  €  <  O.lrad^ii  6° 


(55) 


with  e  in  radians. 

Assuming  the  separation  base  L  =  2  feet,  ai  =  a2  =  45°,  and  the  two 
cameras  are  pointed  symmetrically  with  respect  to  the  perpendicular  on  the 
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L 


Figure  10;  Two  camera  FOV  overlap 


base  Lj  then  Table  4  shows  the  largest  possible  distances  d  for  a  given  FOV 
overlap  e. 


£ 

Pi 

P2 

d 

Total  FOV 

1° 

-22° 

22° 

114.6  ft 

89P 

2° 

1 

to 

o 

o 

o 

o 

5T3  ft 

88P 

Table  4;  FOV  intersection  distance  for  various  levels  of  overlap,  separation  base  L  =  2 
feet,  FOV=45° 


A  minimum  overlap  of  2°  (about  5%  of  each  camera^s  FOV)  is  recom¬ 
mended  to  have  a  smooth  transition  of  the  tracks  between  the  cameras. 
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Appendix  F  Previous  Measurement  Extrac¬ 
tors  Considered 


Appendix  F.l  Measurement  Extractor  1 

The  measurements  are  extracted  from  each  frame  of  data  by  first  calculating 
the  average  pixel  intensity  over  a  sliding  window  and  using  it  in  a  threshold 
to  declare  a  detection  in  the  center  of  the  window.  Each  pixel  of  the  frame 
can  be  divided  by  the  appropriate  average  value  of  its  neighborhood  to  find 
pixels  which  stand  out  from  their  neighborhood.  The  neighborhood  average 
for  pixel  z,  /j,  will  be  calculated  as 


li 


1 

L1L2 


E  A 

weWi 


(56) 


where  4^  is  the  intensity  of  pixel  w,  L1L2  is  the  size  of  the  window,  and  Wi 
is  the  set  of  pixels  of  the  sliding  window  centered  at  pixel  i.  Pixel  i  will  then 
be  declared  a  detection  if 

J>T  (57) 

where  4  is  the  intensity  of  pixel  i  and  r  is  the  set  threshold.  This  is  Extractor 
la. 

The  detected  pixels  are  grouped  and  the  centroid  of  each  group  calculated 
as  described  is  section  2.1. 

This  method  of  measurement  extraction  alone  may  result  in  a  large  num¬ 
ber  of  false  measurements  if  areas  of  the  frame  have  especially  low  pixel 
intensities.  Figure  11  shows  the  centroid  measurements  from  the  extractor 
which  uses  a  threshold  of  1.35  and  Li  =  L2  =  21.  The  large  number  of 
measurements  at  the  top  of  the  frame  are  a  result  of  the  low  pixel  intensities 
which  can  easily  exceed  the  set  threshold  without  being  much  larger  than  the 
neighborhood  average  intensity.  In  order  to  avoid  this  behavior,  the  detec¬ 
tion  rule  can  be  augmented  by  requiring  a  detected  pixel  to  also  exceed  the 
average  pixel  intensity  of  the  entire  frame.  This  is  Extractor  lb.  Figure  12 
shows  the  centroid  measurements  from  this  updated  measurement  extractor. 
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Figure  11:  Extractor  la  with  threshold  of  1.35 


Appendix  G  Previous  Motion  Models  Con¬ 
sidered 

Appendix  G.l  Wiener  Process  Acceleration  Model 

Originally,  a  discrete  Wiener  process  acceleration  (WPA)  model  [2]  was  used, 
where  the  state  equation  is 

x{k  +  1)  =  Fx{k)  +  rt;(^)  (58) 
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Figure  12:  Extractor  lb  with  full  frame  average  exceedance  rule 


where  x{k)  is  the  state  at  time  k  and  v{k)  is  the  process  noise  during  the  kth 
sample  (acceleration  increment  in  this  case),  and 
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where  T  is  the  sampling  time  (T  =  1  /30  in  our  case). 

The  standard  deviation  of  the  measurement  noise  was  assumed  to  be 
0.5  pixels,  while  the  standard  deviation  of  the  process  noise  (acceleration 
increment  in  a  sampling  interval)  was  chosen  to  be  ^/b  pixels/s^. 


Appendix  H  Previous  Track  Initialization  Schemes 
Considered 

Appendix  H.l  Track  Initialization  1 

A  validation  gate  (in  pixel  space),  based  on  the  assumed  maximum  velocity 
of  a  target  over  one  sampling  period,  was  set  up  for  each  measurement  bear 
tion  for  the  potential  tracks.  In  this  case,  a  gate  which  assumed  a  maximum 
velocity  of  100  pixels  in  the  x-direction  and  50  pixels  in  the  ^-direction  was 
used.  Tracks  were  confirmed  and  initialized  when  a  second  consecutive  mea¬ 
surement  is  found  within  the  validation  gate  of  the  potential  tracks.  These 
tracks  were  then  added  to  the  list  of  currently  running  tracks  and  any  po¬ 
tential  tracks  without  a  second  measurement  in  their  validation  gates  were 
dropped  from  the  list  of  potential  tracks. 


Appendix  I  Multiple  Camera  Registration 

Appendix  I.l  Relative  Registration 

In  order  to  track  targets  using  multiple  sensors,  the  sensors  must  be  properly 
registered  in  order  to  combine  the  data.  For  this  passive  collision  avoidance 
system,  the  cameras  must  be  registered  relative  to  each  other  at  a  minimum, 
i.e.,  it  is  only  necessary  to  know  the  difference  of  the  azimuth  and  elevation 
angles  of  each  camera  and  not  the  actual  azimuth  and  elevation  angles.  Ab¬ 
solute  registration  (section  Appendix  1.2)  is  only  necessary  for  evaluating  the 
performance  of  the  tracker  with  a  target  of  known  location  (e.g.,  one  with 
GPS  data). 

In  order  to  estimate  the  difference  in  azimuth  angles  of  two  overlapping 
cameras,  the  measurements  of  a  target  within  the  overlapping  portion  of  the 
camera  frames  can  be  used.  The  relation  between  the  azimuth  measurements 
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of  two  cameras  (e.g.,  camera  A  and  camera  B)  is 

aB^dLB=OLA-\-^A  (61) 

where  is  the  boresight  azimuth  of  camera  x  and  is  the  measured  az¬ 
imuth  to  the  target  (relative  to  the  focal  plane  of  camera  x,  and  available 
from  (27)  in  Appendix  A).  Rearranging  (61)  results  in  the  difference  of  the 
camera  boresight  azimuth  angles  (a): 

a  =  otB  —  ^A  (62) 

A  similar  equation  holds  for  the  elevation  angle. 

Additionally,  a  possible  non-zero  roll  angle  in  either  camera  must  be  taken 
into  account.  The  measured  azimuth  and  elevation  angles  can  be  corrected 
by  using  the  transformation  T3  from  Appendix  B.  This  will  correct  the 
measured  azimuth  and  elevation  angles  for  the  roll  angle  rotation  of  the 
camera  (assumed  known)  so  that  the  new  azimuth  and  elevation  angles  are 
relative  to  a  camera  with  zero  roll  angle. 

The  boresight  angle  difference  (62)  can  be  calculated  for  each  measure¬ 
ment  within  the  overlap  of  the  two  camera  and  this  data  can  be  used  in  a  LS 
(least  squares)  estimator.  The  LS  estimator  in  this  case  simply  corresponds 
to  the  sample  mean  of  the  calculated  boresight  angle  differences. 

Appendix  L2  Absolute  Registration 

Registering  one  camera  in  relation  to  another  will  provide  the  difference 
in  azimuth  and  elevation  angles  of  the  boresight  of  each  camera.  Using 
this  information,  if  the  pointing  direction  of  one  camera  is  known,  then  the 
pointing  angle  of  the  second  camera  is  simply  the  pointing  angles  of  the  first 
plus  the  differences  obtained  from  relative  registration. 

In  order  to  determine  the  pointing  angle  of  the  first  camera,  available 
GPS  information  for  a  tracked  target  can  be  used.  The  GPS  data  can  be 
converted  to  ENU  coordinates  by  using  the  calculations  of  Appendix  C  with 
the  origin  of  the  ENU  coordinate  system  as  the  camera  location.  The  ENU 
coordinates  can  then  be  used  to  determine  the  azimuth  and  elevation  angles 
from  the  camera  to  the  target.  The  measurements  of  the  target  in  the  focal 
plane  (in  pixels)  can  be  translated  to  azimuth  and  elevation  angles,  relative 
to  the  focal  plane,  by  using  the  equations  of  Appendix  A.  The  boresight 
azimuth  angle  is  then 

o^BS  =  otQPs  —  ocpp  (63) 
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where  ass  is  the  boresight  azimuth,  aops  is  the  azimuth  obtained  from  the 
ENU  coordinates  of  the  GPS  data,  and  app  is  the  azimuth  of  the  measure¬ 
ment  relative  to  the  focal  plane.  A  similar  equation  can  be  used  to  solve  for 
the  elevation  angle.  Multiple  points  of  GPS  data  can  then  be  used  in  a  LS 
estimator  to  determine  the  estimated  boresight  angle  of  the  camera. 

If  GPS  data  is  available  for  a  target  in  the  second  camera,  the  same  equa¬ 
tions  can  be  used  to  determine  the  boresight  angles  of  the  second  camera. 
The  boresight  angles  of  the  second  camera  can  also  be  calculated  using  the 
data  from  relative  registration  along  with  the  boresight  angles  calculated  for 
the  first  camera.  These  two  sets  of  boresight  angles  can  then  be  compared. 
Additionally,  the  GPS  data  for  the  target  in  the  second  camera  can  also  be 
compared  to  the  angles  calculated  from  relative  registration  and  the  bore¬ 
sight  angles  of  the  first  camera.  Table  5  shows  the  boresight  angle  estimates 
of  Camera  A  calculated  both  directly  from  GPS  data  and  also  indirectly 
from  the  relative  registration  of  Camera  A  to  Camera  B  (using  the  bore¬ 
sight  of  Camera  B  calculated  from  GPS  data).  Additionally,  the  azimuth 
and  elevation  angles  to  the  target  were  calculated  using  the  relative  registrar 
tion  boresight  angles,  and  the  RMS  errors  were  then  calculated  (using  the 
azimuth  and  elevation  to  the  target  calculated  fi’om  the  GPS  data  as  truth). 


Relative  Registration 

1  GPS 

RMS  error 

Azimuth 

38.5° 

38.68° 

0.808° 

Elevation 

19.76° 

00 

o 

1.578° 

Table  5:  Comparison  of  boresight  calculations  (for  Camera  A)  resulting  from  relative 
registration  to  those  resulting  from  direct  GPS  data  estimation 
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The  following  are  representative  frames  of  the  process  video  created  by  UConn.  The  red  track 
indicates  that  their  measurement  extractor  has  located  the  target  aircraft  and  has  started  a 
track.  In  Figure  59,  the  angular  velocity  (degrees/second)  of  tracked  targets  is  displayed.  The 
GPS  tracking  information  in  on  the  left  and  on  the  calculated  from  the  image  is  on  the  right. 


Figure  57:  Video  track  of  Uitraiight  of  Figure  6  of  Figure  58:  Video  track  of  Uitraiight  of  Figure  7  of 
the  UConn  report  showing  no  coliision.  the  UConn  report  showing  coliision  course. 


Figure  59:  Video  track  of  Piper  aircraft  using  side  by  side  L3  cameras  at  Pottstown  Airport  (Figure 
28).  Left  hand  side  in  the  data  logger  GPS  track,  right  hand  side  is  the  calculated  track 


5.2  Image  Differencing 

While  UConn  focused  on  single  frame  detector,  with  the  issue  of  clutter  removal  and  S/N,  using 
image  differencing  is  another  approach.  By  differencing,  clutter  would  be  “zeroed”  out  because 
it  is  static  frame  to  frame  and  targets  which  are  moving  would  create  additional  contrast  against 
the  static  background.  This  concept  is  illustrated  in  Figure  60,  the  algorithmic  flow  is  illustrated 
in  Figure  62.  The  bird  flying  in  from  the  right  is  not  apparent  in  the  image  on  the  left.  The 
background  including  the  ground  clutter  goes  to  zero.  However,  image  differencing  is  very 
susceptible  to  motion  of  the  platform  during  collection.  The  image  must  be  stabilized  to  be 
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effective  (Figure  61).  Note:  at  this  time  due  to  the  ending  of  the  program  stabilization  was  not 
completed. 
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Figure  62:  Flow  diagram  of  the  image  through  the  sensor. 


6  Hardware  Approach  for  Tier  I  UAV 

MilSys  Technologies  as  been  in  contact  with  a  few  of  the  mini-UAV  manufacturers  to  discuss 
our  PCAS  system.  Our  existing  configuration  of  a  multi-sensor  system  does  not  meet  their 
SWAP  requirements.  What  has  come  out  of  our  discussions,  is  that  the  mission  profile  is 
different  enough  from  the  Predator  class,  that  some  compromises  on  performance  can  be  made 
to  reduce  the  SWAP  requirements. 


Lens  &  FPA 


Ribbon  Cable 


IMU  Daughter 

FPGA/DSP 

- - H 

Figure  63:  Side  on  diagram  of  the  propose  sensor  /  TP.  Note  the 
dimensional  scales. 


Weight 

3600AS 

3oz 

w/lens 

FPGA/ 

DSP 

2oz 

IMU 

2oz 

Total 

7oz 

Tables:  Estimated 
weight. 

As  discussed  in  Section  1 .3,  the  most  import  factor  in  avoiding  collisions  is  the  time  to  collision. 
For  the  Predator  class  under  most  situations,  it  has  been  specified  as  32  seconds  [minimum]. 
Due  to  the  differing  mission  scenarios  and  platform  design  for  the  mini-UAS,  this  time  can  be 
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reduced  to  «1 0  seconds.  MITL  time  goes  to  0  and  the 
bank  angle  can  be  significantly  >18°  reducing  the  time  to 
«5  seconds.  Detection  and  tracking  time  remains  constant 
at  2-5  seconds.  At  a  250knot  head  on  closing  speed,  10 
seconds  is  1 .5km  minimum  slant  range  at  detection. 

Microbolometer  technology  is  more  than  adequate  for  this 
mission. 

The  innovative  approach  is  integrate  COTS  IR  technology, 
such  as  the  L-3  Thermal  Eye  3600AS  (Table  9),  Xilinx 
FPGA/DSP  with  an  IMU  currently  under  development  at 
MilSys  Technologies.  Figure  63  is  a  side  on  view  of  what 
the  hardware  would  approximately  look  like.  The  lens  and 
FPA  are  separate  from  the  control  electronics  for  the 
maximum  mounting  flexibility  (Figure  63).  Table  8  is  the 
estimate  of  the  total  weight  «7oz  (<0.5lbs).  The  Scan 
Eagle  published  payload  is  6lbs  the  Silver  Fox  is  5lbs.  So  the  proposed  configuration  is  at 
«10%.  With  additional  engineering  one  or  two  of  the  PCBAs  could  potentially  be  eliminated. 
The  major  weight  driver  is  the  lens. 


Optics 

LO 

O 

V 

FOV 

cn 

o 

o 

Pixels 

160 X 120 

Pixel  Size 

30pm 

Technology 

a-Silicon 

Wavelength 

7-1 4pm 

Sensitivity 

<50mK 

Table  9:  L-3  3600AS. 

Figure  64:  Block  diagram  of  how  the  new  functions  of  air-ground  would  be  implemented  to 
leverage  the  existing  PCAS  architecture  and  developments. 

With  only  one  sensor,  complex  image  transformation  will  not  be  required  by  the  TP,  however, 
the  new  function  of  air-ground  will  replace  it.  The  block  diagram  of  the  information  is  more 
complex  with  the  addition  of  the  air-ground  detector  (Figure  64). 

The  lens  currently  offered  by  L-3  on  the  3600AS  has  a  FOV  of  53°  X  37°.  Ideally  for  a  forward 
looking  only  EO/IR  SAA  system,  a  horizontal  FOV  of  90°-120°  would  be  optimal.  However, 
microbolometers  need  a  low  F/#  lens,  this  drives  weight  up.  An  innovative  aspect  of  this 
proposal  is  a  contrarian  approach  to  lens  design.  Most  lenses  are  designed  for  imaging.  A 
trade  study  will  be  performed  looking  at  several  alternative  approaches:  For  example,  modified 
off  the  shelf  single  element  (Figure  65)  similar  to  the  Temmek  Optics  IRViper.  Note  that  the 
weight  is  almost  4X  that  of  the  proposed  camera.  LWIR  transmitting  plastics  are  available 
(Figure  68).  Designs  will  be  reviewed  that  would  be  a  combination  of  aspheres,  fresnel  lenses 
(Figure  67),  and/or  diffractive  lenses  molded  out  of  plastic  or  chalcogenide  glass.  A  final  design 
possibility  is  a  complex  anamorphic  wide  angle  view  converter  (Figure  66).  The  optical  system 
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will  have  a  predetermined  optical  power  in  the  horizontal  axis  different  from  the  vertical  axis. 
Typically,  a  pair  of  prisms  is  used  to  form  an  anamorphic  system.  For  this  system,  the  prism 


Figure  65:  Temmek  Optics  -  IRViper  Figure  66:  Schematic  of  a  wide  angie  FOV 
Lens,  F/#  0.8,  FOV  «80°  X  60°,  Weight  -  converter. 

260g.“™ 


4  6  8  10  12  14  16 

Wavelength  (pm) 


Figure  67:  Schematic  of  a  Fresnei  Figure  68:  Transmission  spectrum  of  Fresnei 

iens.***'''  Technologies  POLY  IR®  2  IR  piastic."’'*''' 

pair  will  be  incorporated  into  a  two  or  three  element  toroidal  lens  system  to  change  the  field  of 
view  from  a  4x3  format  for  the  microbolometer  sensor  to  a  different  field  of  view  format.  That  is, 
to  use  all  of  the  pixels  most  efficiently  for  a  4x3  image  format  the  corresponding  field  of  view 
would  be  40deg.  x  30  deg.  As  an  example,  to  change  to  a  field  of  view  to  60  deg.  or  1 20  x  30 
deg.,  the  toroidal  lenses  would  need  to  compress  the  horizontal  axis  by  a  factor  of  1 .5  for  the 
same  sensor.  The  resultant  image  will  have  an  image  which  is  2.25  to  4.5X  times  brighter  in  the 
vertical  FOV  than  the  horizontal  FOV.  The  overall  tradeoffs  for  the  entire  system  will  be  the 
lowest  f/number  that  can  be  achieved  while  holding  the  distortion  and  the  allowable  vignetting  at 
the  edges  of  the  field  to  a  minimum,  while  optimizing  the  lens  system  for  weight,  cost  and 
volume. 


7  Extension  of  the  Technological  Approach  to  solve  the  full  CA 
problem 
7.1  Introduction 

As  defined  by  the  JIPT’'’'’'''  in  their  recently  published  draft  UAS  requirements  document,  a  sense 
and  avoid  (SAA)  system  encompass  the  following  functions:  air-to-air,  air-to-ground,  ground-to- 
ground,  ground-to-air,  and  man  made  hazards  (Table  10).  Part  of  the  mission  planning  is  an 
environmental  forecast  encompassing  both  atmospheric  phenomena  and  manmade  hazards 
(i.e.  smoke).  Flowever,  this  forecast  is  for  the  macro  airspace  and  may  not  necessarily  be 
accurate  for  the  localized  mission  airspace.  As  long  as  the  UAV  remains  within  the  line  of  sight 
of  the  warfighter,  it  has  a  SAA  system.  As  soon  as  the  UAVs  fly  out  of  the  line  of  sight  on  the 
battle  field,  risks  to  the  mission  increase.  Adding  the  capability  to  sense  the  environment  would 
enhance  the  utility  of  UAVs,  by  allowing  more  autonomous  operation,  provide  localized 
forecasting  for  battlefield  management,  and  provide  real-time  feedback  to  the  onboard  ISR 
sensor  suit  to  maintain  probability  of  detection,  contrast  level,  etc.  This  capability  would  reduce 
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the  workload  of  the  warfighter,  increase  mission  success  rate,  and  expand  mission 
opportunities. 

To  address  these  needs,  various  sensors  concepts  based  on  the  microbolometer  were 
developed.  The  ability  to  differentially  sense  temperature,  passive  ranging,  IR  spectrum  of 
clouds  and  polarization  were  reviewed  for  their  applicability  to  potentially  solve  these  issues  as 
well  as  the  technological  feasibility. 


Phase  of  FlighM 

nt 

□ 

Rationale  /  Ampiificat 

Draft  Threshold 

1.4  UAS  Operations 

■ 

□ 

All  Phases 

1.4 

The  UAS  shall  be  capable  of  avoiding  hazards. 

Overarching  Hazard  Avoidance  requirement. 

All  Phases 

1.4.1 

The  UAS  shall  be  capable  of  avoiding  coiiision  hazards. 

The  UAS  initiates  an  avoidance  maneuver  base 
collision  avoidance  system  determination  of  a 
potential  hazard. 

All  Ground 

1.4.1. 1 

While  on  the  ground,  the  UAS  shall  be  capable  of  avoiding 
airborne  hazards  (Ground-to-Air). 

The  requirement  is  to  avoid  becoming  a  runway  incursion,  as  in 
clearing  final  before  taking  the  active  runway. 

All  Ground 

1.4.1. 2 

While  on  the  ground,  the  UAS  shall  be  capable  of  avoiding 
ground  hazards  (Ground-to-Ground). 

During  taxi  operations  in  the  airfield  environment,  the  UAS  must 
be  able  to  avoid  fixed  obstacles,  other  ground  aircraft,  as  well  as 
ground  support  equipment.  During  takeoff,  the  UAS  shall  be 

Rxed  obstaoles  2015 

Table  10:  JIPT  SAA  requirements. 


7.2  Cloud  Detection  via  Infrared  Imaging  (Background) 

Clouds  can  be  detected  on  an  airborne  platform  with  LWIR  infrared  imaging  owing  to  the  high 
radiometric  contrast  between  the  clear-sky  emission  and  the  cloud  emission’*'’’*'''.  The  Optical 
Remote  Sensors  Laboratory  (ORSL)  at  Montana  State  University,  under  the  direction  of  Dr. 
Joseph  Shaw,  has  developed  instruments  and  algorithms  for  detecting  clouds,  measuring  cloud 
statistics,  and  characterizing  cloud  type  using  ground-based,  uncooled  microbolometer  thermal 
infrared  cameras’*’*’*''';’*’*’*''",’*’*’"'"'.  These  techniques  were  optimized  for  vertically  viewing  ground- 
based  infrared  imagers,  could  be  modified  to  operate  in  an  airborne  environment  for  detecting 
clouds  at  angles  other  than  vertical. 

Figure  69  shows  the  down-welling  atmospheric  spectral  radiance  calculated  for  the  1976  U.S. 
Standard  Atmosphere,  plotted  versus  wavelength  for  a  wavelength  range  of  5-18  ).im.  This 
figure  illustrates  the  atmospheric  emission  spectrum  that  would  be  seen  by  a  ground-based, 
upward-viewing  infrared  sensor.  The  bottom  three  curves  are  for  a  clear  sky  with  water  vapor 
contents  that  increase  upward  from  the  bottom  curve:  0.1  x,  lx,  and  2x  the  normal  water  vapor 
content.  The  top  curve  is  for  a  cloudy  sky  with  mid-level  altostratus  clouds.  Figure  70  is  the 
vertical-path  atmospheric  transmittance  plotted  versus  wavelength  for  the  1976  U.S.  Standard 
Atmosphere  model,  with  1  x  water  vapor  (top)  and  2x  water  vapor  (bottom).  These  figures  show 
that  the  atmosphere  exhibits  high  transmittance  where  there  is  low  emission,  and  vice  versa. 
This  is  the  result  of  emission  arising  proportional  to  absorption. 

Examining  Figure  69  and  Figure  70  within  the  spectral  bandwidth  of  most  LWIR  cameras  (~8- 
14  i^m)  reveals  several  important  points  about  cloud  detection  with  IR  cameras:  1)  the 


81 


atmosphere  has  relatively  high  transmittance,  especially  with  low  water  vapor  content);  2)  Many 
clouds  are  quite  bright  relative  to  the  clear-sky  radiance;  and  3)  variations  in  atmospheric  water 
vapor  can  be  interpreted  as  thin  clouds  unless  the  intervening  atmospheric  emission  is 
compensated  for  carefully.  The  difficulty  of  distinguishing  between  thin  cirrus  clouds  and  water 
vapor  variations  is  further  illustrated  by  Figure  71,  which  plots  down-welling  atmospheric 
emission  versus  wavelength  (similar  to  Figure  69)  for  a  clear  sky,  a  very  thin  cirrus  cloud,  an 
altostratus  cloud,  and  a  cumulus  cloud.  Comparing  Figure  69  and  Figure  71  shows  that  in  this 
case  the  thin  cirrus  emission  is  less  than  the  radiance  change  that  would  occur  with  a  doubling 
of  water  vapor. 


Figure  69:  Down-welling  atmospheric  emission 
spectrum  for  the  Atmosphere  model,  shown  for  a 
clear  sky  with  three  different  values  of  water  vapor 
content  (0.1  x,  1  x,  and  2x  the  standard  value)  and 
for  altostratus  clouds  (top  blue) 


Figure  70:  Atmospheric  transmittance  for  a  zenith 
path  through  the  Standard  Atmosphere.  The  top 
(blue)  curve  1  x  water  vapor,  and  the  bottom  curve 
(red)  is  for  2x. 


In  ground-based  cloud  measurements  for  climate  studies  it  is  critical  to  detect  even  very  thin 
cirrus.  Therefore,  these  situations  are  handled  through  the  use  of  IR  cameras  that  are  calibrated 
very  carefully  to  achieve  radiometric  measurements  with  high  accuracy  and  stability’'’'’^''',’'’'’'''"'.  In 
these  situations  the  clear-sky  emission  is  calculated  using  ancillary  data  that  provides  a 
measure  of  the  precipitable  water  vapor  and  near-surface  air  temperature,  and  this  emission  is 
subtracted  from  the  image  to  generate  a  “residual  radiance”  image.  These  residual  radiance 
images  are  ideally  zero  everywhere  except  where  clouds  exist.  The  magnitude  of  the  residual 
radiance  is  used  to  determine  cloud  type.  An  example  of  this  type  of  data  is  shown  in  Figure  72. 
In  this  figure  the  left-hand  panel  is  the  calibrated  radiance  sky  image,  the  center  panel  is  the 
residual  radiance  image,  and  the  right-hand  panel  is  the  cloud-detection  image  color  coded  by 
cloud  type  (dark  blue  for  clear,  red  for  the  radiometrically  brightest  clouds,  and  other  colors  for 
intermediate  cases). 
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Atmospheric  Emission  with  Clouds 


10  11  12 
Wavelength  (pm) 


Figure  71 .  Down-welling  emission  spectrum  for 
clear  sky  (blue  curve,  bottom),  very  thin  cirrus  at  9 
km  altitude  (red  curve,  2"*^  from  bottom),  alto- 
stratus  cloud  at  2.4  km  (green  curve,  3^'*  from 
bottom),  and  cumulus  cloud  with  its  base  near  100 
m  (teal  curve,  top). 

In  the  proposed  technology,  the  emission  signatures  will  change  significantly  from  what  is 
shown  in  the  preceding  figures,  primarily  because  of  the  large  range  of  viewing  angles  to  be 
encountered.  Also,  we  are  not  likely  to  have  sufficient  information  available  to  properly  estimate 
the  emission  of  the  intervening  atmosphere  (which  requires  a  reasonable  estimate  of  the 
atmospheric  temperature  and  water  vapor  profiles,  viewing  angle,  and  range  to  the  cloud). 
Furthermore,  because  of  size  and  weight  constraints,  the  imager  likely  will  not  be  deployed  with 
onboard  calibration  sources,  making  the  direct  use  of  previously  developed  algorithms 
insufficient  for  this  application.  Nevertheless,  the  principles  are  the  same  and  the  need  to 
accurately  characterize  thin  clouds  is  not  a  necessary  component.  Consequently,  we  propose  to 
develop  algorithms  that  operate  on  a  more  relative  basis,  identifying  clouds  through  the  use  of  a 
radiometric  calibration  that  is  measured  in  the  laboratory  and  applied  to  the  imager  in  flight. 
Shaw’s  group  at  Montana  State  University  has  recently  patented  algorithms  that  allow  very 
accurate  calibration  of  infrared  imagers  without  onboard  blackbody  sources,  and  similar 
techniques  will  be  used  here. 

The  algorithms  to  be  developed  would  will  rely  less  on  the  absolute  radiance  (which  is  a 
necessary  component  of  the  cloud  imaging  work  done  at  Montana  State  University  for  climate 
studies)  and  instead  rely  more  on  the  relative  variations  of  brightness  seen  while  the  air  vehicle 
is  flying.  Measurements  of  vehicle  altitude  and  attitude  will  be  used  to  determine  the  nominal 
pointing  angle  of  each  pixel,  and  algorithms  will  be  developed  that  use  either  look-up  table  or 


Calibrated  Radiance  Residual  Radiance  Cloud  Detection 


Figure  72.  Images  from  the  wide-angle  Infrared  Cloud  Imager  system  developed  at  Montana  State 
University:  (left)  radiance,  (center)  residual  radiance  after  removing  atmospheric  emission,  and  (right) 
detected  clouds  classified  by  cloud  type  (from  0  =  no  clouds  to  5  =  thick  clouds  as  described  in  text). 
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curve-fit  routines  for  estimating  the  background  atmospheric 
signature  as  a  function  of  viewing  angle.  This  type  of  approach  is 
not  likely  to  offer  sufficient  stability  and  accuracy  to  allow  reliable 
detection  of  thin  cirrus,  but  in  this  application  the  much  more 
important  clouds  to  detect  are  thicker  clouds  that  indicate  the 
presence  of  stormy  weather  or  total  obstruction  of  the  view  in  a 
particular  direction.  As  shown  in  Figure  69,  these  clouds  are  very 
easy  to  detect  because  of  their  large  radiometric  brightness  relative 
to  the  clear  atmosphere. 

Additional  information  may  be  available  from  the  spectral 
distribution  of  radiance  within  the  camera’s  full  bandwidth  as  is 
being  developed.  As  is  indicated  in  Figure  69,  there  is  a  distinctly  different  slope  of  radiance 
versus  wavelength  for  clear-sky  emission  and  cloud  emission.  Satellite  imager  algorithms 
sometimes  rely  on  this  fact  to  detect  clouds  through  a  “split-window”  technique  or  similar 
approach  relying  often  on  ratios  of  the  radiance  detected  at  two  different  wavelengths’'’'’''^ 

7.3  LWIR  Multispectral  Sensor 

The  literature  in  the  atmospheric  science  journals  is  limited.  Clouds  can  be  detected  on  an 
airborne  platform  with  LWIR  infrared  imaging  owing  to  the  high  radiometric  contrast  between  the 
clear-sky  emission  and  the  cloud  emission’'’'’^'''.  The  Optical  Remote  Sensors  Laboratory  (ORSL) 
at  Montana  State  University,  under  the  direction  of  Dr.  Joseph  Shaw,  has  developed 
instruments  and  algorithms  for  detecting  meteorological  phenomena  (i.e.  clouds,  measuring 
cloud  statistics,  and  characterizing  cloud  type)  using  ground-based,  uncooled  hyperspectral 
microbolometer  based  systems'‘'’'‘''’'‘'".  Another  recent  paper  by  Tamachi’*""  et  al.  evaluated  the 
potential  of  a  ground-based  IR  camera  (FLIR  Systems,  Inc.)  to  map  cloud  base  temperatures 
around  supercell  storms.  This  study  included  a  nice  summary  of  the  difficulties  in  acquiring 
accurate  measurements  from  the  ground  because  of  IR  sources  from  water  vapor,  aerosols, 
and  dust  within  the  atmospheric  boundary  layer.  For  a  UAV  flying  above  the  turbid  boundary 
layer,  improved  IR  measurements  could  be  obtained,  free  from  the  artifacts  described  for 
ground-based  imagers  as  described  by  Tamachi  et  al. 

What  has  been  investigated,  is  the  possibility  of  combining  a  SAA  system  that  is  capable  of 
providing  both  an  environmental  sensing  and  air-air  collision  avoidance  capability.  At  the  heart 
of  the  MilSys  Technologies  PCAS  are  microbolometers  which  are  low  cost  SWAP  efficient  LWIR 
sensors.  As  can  be  seen  in  Figure  73-Figure  77,  both  clouds  and  an  aircraft  are  visible  in  the 
images. 


Figure  73:  LWIR  image  of 
clouds  using  libolometer. 
Red  circle  is  an  aircraft. 


Figure  74:  Image  Figure  75:  Image  Figure  76:  Cloud  Figure  77:  Same  day 

before  rain.  during  rain.  formations.  as  Figure  76 

Microbolometers  are  monochromatic  detectors  so  that  the  radiant  extent  of  an  object  is  reduced 
to  intensity.  Therefore,  a  hot  and  distant  object  could  have  the  same  intensity  as  a  close  and 
cool  object.  The  larger  the  radiance  differential  is  between  meteorological  phenomena  and  the 
background,  the  greater  the  probability  that  this  phenomenon  is  a  hazard  and  should  be 
avoided.  Dr.  Shaw  has  developed  algorithms  that  can  be  used  to  detect,  track,  and  threshold 
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Figure  81 .  An  example  of  a  fabricated  guided-mode  resonance  element  with  comparison  between 
experiment  and  theory  for  a  double-layer  device  (a).  The  parameters  used  for  the  theoretical  curve 
fit  are  close  to  the  nominal  values;  they  are  cover  refractive  index  nc=1.0,  n1  =1.454  (Si02), 
n2=1.975  (Hf02),  substrate  index  ns=1.454,  d1=135  nm,  fill  factor  (fraction  of  period  occupied  by 
the  higher-index  medium)  F=0.58,  d2=208  nm,  period  FA=446  nm,  and  angle  of  incidence  6=0°  (b). 
A  scanning  electron  micrograph  (SEM)  and  a  schematic  of  the  device  are  shown  (c) 

meteorological  phenomena.  Other  researchers  have  developed  algorithms  to  autonomously 
avoid  and  plan  a  path  around  them’*"''.  However,  the  lack  of  discrimination  and  ranging  for 
intensity  only  sensing,  makes  path  planning  and  avoidance  more  difficult.  Even  with  these 

limitations,  this  system  concept  will  work  and  can  be  integrated  with  the  current  development  of 

our  PCAS. 


The  innovative  technology  is  to  create  a  multispectral  microbolometer.  This  will  allow  the 
discrimination  of  meteorological  phenomena  by  allowing  its  radiant  extent  to  be  measured. 
These  measurements  will  be  used  in  three  ways  (3):  as  raw  wavelength  data,  converted  to 
differential  temperature  and/or  range.  The  result  will  be  to  reduce  false  positives,  improve  path 
planning  and  avoidance,  and  the  detection  and  identification  of  meteorological  phenomena  etc. 
This  will  be  accomplished  by  developing  and  integrating  three  (3)  technologies;  guided  mode 
resonate  (GMR)  filters,  microbolometers,  and  a  moving  FPA. 

Guided-mode  resonance  (GMR)  filters  consist  of  fine  spatial  patterns  arranged  to  control  the 
propagation  of  light  (Figure  81).  Lithographic  patterning  of  dielectric  surfaces,  layers,  or  volume 
regions  yields  low-loss  structures  that  affect  the  spatial  distribution,  spectral  content,  energy 
content,  polarization  state,  and  propagation  direction  of  an  optical  wave.  GMR  filters  can  exhibit 
a  filter  function  that  exceeds  the  performance  found  with  50  to  1 00  layer  thin-film  filters.  A 
significant  body  of  published  work  has  focused  on  narrow-line  GMR  filters  which  have  been 
demonstrated  both  theoretically  and  experimentally’*'''  ’*'''"'.  These  filters  require  tight  fabrication 
tolerances.  The  key  insight  of  the  proposed  technology  is  to  use  GMR  filters  in  a  contrarian 


Figure  78: 
Representation  of 
different  filters. 


horizontal-polarized  diagonal-polarized 


Figure  79:  Schematic  of  a  GMR  Figure  80:  SEM  of  a  wire  grid 
membrane  (purpie)  above  a  pixei  poiarizer  produced  by  L-3. 
(iight  biue). 
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approach.  That  is  as  a  broadband  filter  (1-3pnn  bandwidth)  with  loose  fabrication  tolerances.  By 
slightly  changing  the  structure  without  changing  the  material  thickness,  the  spectral  selectivity 
properties  can  be  modified.  A  multispectral  sensor  can  then  be  constructed  out  of  several 
different  GMR  designs  with  a  minimum  of  fabrication  steps  (Figure  78). 

The  temperature  range  most  commonly  be  encountered  by  the  multispectral  microbolometer  will 
be  0-50°C  which  has  a  peak  blackbody  emission  in  the  8-1 2pm  range  (LWIR)  and  which 
corresponds  with  the  peak  sensitivity  of  a  microbolometer.  By  designing  GMR  filters  that  are 
spaced  across  the  LWIR  (e.g.  8-9pm,  9-1 0pm,  10-1 1pm,  1 1-1 2pm  etc.)  the  radiant  extent  of 
meteorological  phenomena  can  be  measured.  By  integrating  the  GMRs  into  the  pixels  of  the 
microbolometer  each  pixel  is  now  wavelength  sensitive  and  permanently  co-registered  (Figure 
78).  This  is  analogous  to  the  Bayer  pattern  used  in  visible  CCD  and  CMOS  cameras. 

Microbolometers  are  thermal  devices  and  their  sensitivity  and  responsivity  are  directly  related  to 
their  thermal  mass.  The  construction  of  the  GMRs  on  the  surface  of  the  microbolometer  would 
drastically  change  their  thermal  mass  requiring  a  substantial  redesign  of  the  entire 
microbolometer.  Independently  and  concurrently,  MilSys  Technologies,  Dr.  Magnusson  and  L-3 
were  working  on  the  concept  of  building  membranes  above  the  surface  of  materials  for  the 
purpose  of  creating  highly  integrated  optical  structures.  L-3  has  developed  the  design,  process, 
and  integration  technology  and  lithography  to  construct  Aluminum  membranes  above  their 
pixels  (Figure  79  &  Figure  80).  These  membranes  are  currently  0.25pm  thick  wire  grid 
polarizers  «1  pm  above  the  surface.  This  technology  will  need  to  be  extended  to  the  design 
parameters  of  the  GMR  filters.  Current  estimates  are  that  the  material  will  be  Silicon  »1  pm  thick 
«3pm  above  the  surface. 


G-^R^G^B-^G 


Based  on  a  fi-ee-floating  sensor  design,  PENTAX SR  is  superior  to  other  sensor-moving  sy  stems  because  it  uses  no 
guide  rails,  allowing  the  sensor  to  oscillate  in  three  directions — horizontally,  vertically,  and  rotationally. 


Bearing  Plate 


Oscillator  Umt 


Figure  82:  Example  of  the  movement  of  Figure  83:  Pentax 

a  FPAto  produce  “multispectral  image 

of  an  IFOV  of  the  FOV.  Each  arrow  (->•) 

represent  the  physical  movement  of  the 

FPA  in  time. 

Many  consumer  based  camera  companies  have  developed  technologies  to  remove  hand  and 
vehicle  vibrations  during  an  image  capture.  They  go  by  the  trade  names  Anti-Shake™,  Vibration 
Reduction™  etc.,  and  have  been  in  wide  scale  production  for  approximately  5yrs.  Camera 
motion  is  detected  via  MEMs  based  gyroscopes  and  compensated  for  by  mechanically  moving 
either  the  lens'*''''  or  the  sensor'  ''  (Figure  83)  These  types  of  inertial  sensors,  mechanical 
mechanisms,  and  feedback  controllers  are  routinely  incorporated  into  D-SLR  cameras  costing 
<$400.  The  mechanical  moving  mechanism  is  usually  a  piezo-electric  or  magnetic  coil  similar 
to  the  driving  mechanisms  in  hard  disk  drives  or  zoom  lenses  on  cell  phones  (Squiggle™)'". 


A  moving  FPA  would  use  this  technology  in  reverse.  Instead  of  removing  vibrations,  the  FPA 

would  be  translated  around  the  image  plane  (Figure  821.  This  motion  would  create  a 
multispectral  image  where  each  instantaneous  field  of  view  (IFOV)  would  be  sampled  with 
several  different  GMR  filters  at  the  frame  rate  of  the  camera.  The  speed  of  these  driving 
mechanisms  is  better  than  5mm/s  (.i.e.  Sms  to  move  25pm  and  accurate  (<1  pm). 
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7.4  LWIR  Temperature  Sensor  and  Passive  Ranging 

At  a  basic  level,  all  radiometric  temperature  sensors  are  single  point  measurements.  This  is 
adequate  when  atmospheric  absorption  can  be  negated.  However  at  a  significant  range,  such 
as  for  this  program,  this  methodology  is  inadequate.  To  use  radiometry  for  temperature 
measurement,  multiple  data  points  (i.e.  two(2)  or  more  data  points  at  differing  wavelength 
bands)  are  required.  Significant  work  has  been  completed,  in  multiband  systems  in  the  MWIR 
and  LWIR'"'.  However,  the  prior  development  work  has  been  focused  on  missiles  and  other 
objects  which  have  significantly  higher  radiance  in  the  MWIR  than  meteorological  phenomena 
(clouds  and  aerosols)  and  these  systems  are  both  SWAP  and  cost  prohibitive  for  UAV  use. 
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a  =  absorption  of  atmosphere 
NEI  =  sensitivity  of  sensor 
L  =  radiance  W  sr/nd 


Equation  1 :  Range  Equation'"' 


Equation  2:  Radiance  form  of  the 
Biackbody  Equation. 
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Equation  3:  The  ratio  between  two(2) 
radiances  of  a  biackbody  is  a  constant 
which  varies  as  a  function  of 
temperature  (7). 


Figure  84:  Biackbody  curve  for  25-1 00°C  (300- 
380K).  The  blue  box  is  the  8-1 0pm  waveband, 
yellow  is  10-1 2pm.  The  ratio  between  the  total 
integrated  radiance  in  each  waveband  indicates 
the  temperature  (Equation  3). 


Radiometrically,  the  range  to  an  object,  in  the  IR,  can  be  determined  if  three(3)  variables  are 
known;  absorption  of  the  atmosphere,  background  temperature,  and  object  temperature 
(Equation  1).  In  general,  only  the  local  background  temperature  is  known  to  any  precision,  the 
atmosphere  can  be  estimated  using  a  priori  information,  and  the  object  temperature  is  unknown. 
To  use  radiometry  to  determine  the  object’s  temperature  (7),  a  minimum  of  two(2)  radiances  (L) 
at  differing  wavelength  bands  are  required  (Figure  84  &  Equation  3). 

These  data  points  are  collected  empirically  by  the  spectrally  sensitive 
sensor.  By  calibrating  it,  a  lookup  table  can  be  competed  to  relate  the 
ratio  of  the  radiances  to  the  temperature  (7)  via  Plank’s  Biackbody 
equation. 

To  accomplish  this  task,  the  multispectral  microbolometer  would  be 
constructed  with  side-by-side  GMR  filters  in  the  8-1 0pm  and  10-1 2pm 
(Equation  4).  The  FPA  would  be  floated  and  translated  back  and  Figure  85-  Data 

forth  as  discussed,  collecting  the  multiband  radiometric  data  for  each  collected  of  an  aircraft 

IFOV  (Figure  18).  Object  temperature  (46;^ )  would  then  be  derived  with  a  COTS 

microbolometer. 
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from  this  data  (Figure  84  &  Equation  3).  Additionally,  if  the  object  is  an  aircraft  and  it  is  being 
tracked,  the  area  around  the  aircraft  could  be  mapped  and  knowing  the  background 
temperature  and  maximum  range  of  the  sensor,  the  absorption  of  the  atmosphere  can  be 
approximated  (Equation  1).  This  approximation  will  be  more  timely  and  accurate  since  it  would 
be  based  on  current  data  instead  of  the  a  priori  data  provided  as  part  of  the  mission  plan. 

7.5  Autonomous  Avoidance 

Given  a  set  of  target  tracks,  the  autonomous  avoidance  algorithms  are  responsible  for 
separation  assurance  and  collision  avoidance.  The  avoidance  algorithm  depends  in  part  on 
available  measurements:  if  range  to  the  target  is  available  (e.g.  through  IR  passive  ranging, 
radar,  ADS-B  or  TCAS)  then  it  may  be  possible  to  employ  more  complex  algorithms  which  seek 
to  minimize  deviation  from  the  nominal  track  while  maintaining  separation.  In  this  case  trajectory 
optimization  methods  such  as  receding  horizon  control  (RHC)  can  be  applied.  RHC  has  been 
used  in  various  UAV  planning  scenarios,  and  recent  work  by  Frew  et  al.'''  was  concerned  with 
adaptively  varying  the  planning  horizon  based  on  the  dynamics  of  the  environment. 

If  range  is  not  available  then  the  avoidance  algorithm  must  control  the  aircraft  to  maintain  non¬ 
zero  bearing  rate  to  targets  (as  discussed  above).  One  approach  is  based  on  potential  field 
methods  for  collision  avoidance.  This  method  was  introduced  by  Khatib''''  and  has  been  used  in 
simulation  and  on  hardware  for  obstacle  avoidance  using  vision  and  inertial  measurements  as 
the  sole  sensors While  potential  field  approaches  can  become  trapped  in  local  minima,  this 
is  generally  restricted  to  cases  where  there  are  many  obstacles  or  narrow  passageways:  this  is 
unlikely  to  occur  in  the  3D  environment  of  air  vehicles.  A  closely  related  approach  based  on 
stream  functions  has  been  shown  to  produce  smoother  trajectories,  and  this  may  be  more 
suitable  for  the  current  application.  Potential  field  and  stream  function  approaches  have  the 
advantage  of  computational  simplicity,  and  thus  will  work  in  real  time. 

7.5.1  Autonomous  avoidance  Algorithms 

Eventhou  the  BAA  was  focused  on  man  in  the  loop  CA  system  some  time  was  invested  in 
reviewing  the  development  of  autonomous  concepts  and  algorithms  and  reaching  out  to  the 
community  on  potential  collaborations.  Most  UAS  manufacturers  and  their  customers  are 
interested  in  a  man-in-the-loop  system  for  controlled  airspace,  but  once  the  UAS  is  en  route  or 
on  station  in  uncontrolled  airspace  where  the  expectation  that  >99.99%  of  the  time  the  airspace 
is  empty  autonomous  avoidance  algorithms  could 
control  or  take  on  an  enhanced  burden  for  de- 
confliction.  Under  these  conditions  CA  is  similar  to 
trajectory  optimization.  Dr.  Jack  Langelaan 
@Penn  State  reviewed  the  literature  and  wrote  a 
short  proposal  on  the  art  of  the  possible. 

Trajectory  optimization  for  UAVs  is  an  active  area 
of  research.  Typical  applications  include  minimal 
altitude  trajectories  for  nap  of  the  earth  flight''^', 
flight  in  urban  environments'^','’",  ground  vehicle 
tracking'’"',  road  following'’"",  target  tracking  and 
state  estimation'’"'',  search'’"',  and  more  recently 
chemical  plume  tracking'’"''. 

Collision  avoidance  trajectories  are  only  required  if 
nominal  trajectories  have  failed  (i.e.  there  is  a  loss 
of  separation  due  to  unforeseen  events  or 
unmodeled  terrain).  Hence  very  rapid  response  is 
required  to  place  the  aircraft  in  a  safe  state  and  the 


Figure  86:  Schematic  representation  of 
the  “safety”  bubbie. 
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response  must  be  computed  in  real  time.  At  the  same  time  the  computational  capability  on 
board  a  mini  UAV  is  likely  to  be  quite  limited,  hence  very  simple  algorithms  are  required. 

One  of  the  simplest  algorithms  for  obstacle  avoidance  is  the  potential  field  approach,  introduced 
by  Khatib'’^''".  A  potential  field  is  computed  based  on  obstacle  position  and  size,  and  the  vehicle 
is  commanded  to  steer  in  the  direction  of  the  local  gradient.  This  approach  has  been  used  for 
simulations  of  UAV  navigation  in  obstacle  strewn  environments  with  vision  and  inertial 
measurements  as  the  sole  sensors'''".  Potential  field  approaches  are  computationally  very 
simple  to  implement,  but  can  become  trapped  in  local  minima  under  some  conditions.  However, 
the  likelihood  of  these  conditions  occurring  in  a  3D  environment  is  unlikely.  In  addition  to 
obstacles,  other  constraints  (such  as  altitude  limitations  or  no-fly  zones)  can  also  be 
incorporated  into  the  potential  field.  The  aircraft’s  goal  position  (or  desired  track)  can  be 
represented  as  a  “sink”  in  the  potential  field. 

Figure  86  shows  a  modification  which  represents  the  potential  field  as  a  “safety  bubble”  around 
the  vehicle.  Any  obstacle  which  enters  the  safety  bubble  causes  a  perturbation  in  the  trajectory 
proportional  to  the  size  of  the  obstacle  and  the  degree  of  bubble  penetration.  The  size  and 
“springiness”  of  the  safety  bubble  depends  on  vehicle  performance  parameters  (e.g.  maximum 
turn  rate)  and  mission  parameters,  and  these  will  be  determined  as  part  of  this  work. 

8  Conclusion 

MilSys  has  demonstrated  the  feasibility  of  using  SWAP  efficient  LWIR  microbolometers  as 
outlined  in  the  Priest  report  circa  1998  as  a  solution  to  the  collision  avoidance  problems  for 
UASs  .  It  has  accomplished  this  by,  modeling  the  response,  collecting  data  and  leveraging  and 
developing  detection  and  tracking  algorithms  that  could  be  implemented  into  future  systems. 
Design  proposals  were  completed  that  showed  the  feasibility  of  extending  the  concept  up  to 
larger  aircraft  and  down  the  smaller  Tier  I  type  UASs.  To  solve  the  entire  collision  avoidance 
problem,  as  discussed  by  the  JIPT,  technology  was  proposed,  that  would  create  an  integrated 
color  FPA  that  could  determine  range,  weather,  and  still  perform  the  collision  avoidance 
function. 
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